2025-10-10T00:35:57.6400421Z Current runner version: '2.328.0' 2025-10-10T00:35:57.6406595Z Runner name: 'i-01dad1d3a0e80f7c0' 2025-10-10T00:35:57.6407282Z Runner group name: 'default' 2025-10-10T00:35:57.6408096Z Machine name: 'ip-10-0-32-50' 2025-10-10T00:35:57.6410386Z ##[group]GITHUB_TOKEN Permissions 2025-10-10T00:35:57.6412624Z Contents: read 2025-10-10T00:35:57.6413168Z Metadata: read 2025-10-10T00:35:57.6413595Z ##[endgroup] 2025-10-10T00:35:57.6415507Z Secret source: Actions 2025-10-10T00:35:57.6416117Z Prepare workflow directory 2025-10-10T00:35:57.6840881Z Prepare all required actions 2025-10-10T00:35:57.6875333Z Getting action download info 2025-10-10T00:35:57.9462801Z Download action repository 'pytorch/test-infra@main' (SHA:264eed5d70b428e3aa5c1a7c98e4330f866e183f) 2025-10-10T00:35:59.5190439Z Download action repository 'pytorch/pytorch@main' (SHA:a6fa4f9c283971c0fb6f60a89674a1f35370ac79) 2025-10-10T00:36:15.4567111Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-10-10T00:36:15.8093756Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-10-10T00:36:16.0606427Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-10-10T00:36:16.2295716Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-10-10T00:36:16.5160463Z Getting action download info 2025-10-10T00:36:16.6303945Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-10-10T00:36:16.8957070Z Getting action download info 2025-10-10T00:36:17.0396471Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-10-10T00:36:17.2191524Z Getting action download info 2025-10-10T00:36:17.3391396Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-10-10T00:36:17.5116519Z Getting action download info 2025-10-10T00:36:17.6484122Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (344e6365a0068c2d2847fcec0c55dd53291d475e) 2025-10-10T00:36:17.6487604Z ##[group] Inputs 2025-10-10T00:36:17.6487990Z build-environment: linux-jammy-py3.10-gcc11-build 2025-10-10T00:36:17.6489918Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-10-10T00:36:17.6492140Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:36:17.6492818Z sync-tag: 2025-10-10T00:36:17.6493825Z timeout-minutes: 240 2025-10-10T00:36:17.6494116Z use-gha: 2025-10-10T00:36:17.6494378Z dashboard-tag: 2025-10-10T00:36:17.6494651Z s3-bucket: gha-artifacts 2025-10-10T00:36:17.6494967Z aws-role-to-assume: 2025-10-10T00:36:17.6495459Z disable-monitor: false 2025-10-10T00:36:17.6495749Z monitor-log-interval: 5 2025-10-10T00:36:17.6496335Z monitor-data-collect-interval: 1 2025-10-10T00:36:17.6496735Z ##[endgroup] 2025-10-10T00:36:17.6497119Z Complete job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:36:17.6985628Z A job started hook has been configured by the self-hosted runner administrator 2025-10-10T00:36:17.7067955Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-10-10T00:36:17.7075357Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:36:17.7075957Z ##[endgroup] 2025-10-10T00:36:18.6933171Z Runner Type: linux.8xlarge.amx 2025-10-10T00:36:18.6933699Z Instance Type: m7i-flex.8xlarge 2025-10-10T00:36:18.6933995Z AMI Name: unknown 2025-10-10T00:36:18.6963677Z AMI ID: ami-08982f1c5bf93d976 2025-10-10T00:36:23.0941566Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-10-10T00:36:23.0941936Z with: 2025-10-10T00:36:23.0942508Z github-secret: *** 2025-10-10T00:36:23.0943001Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-10-10T00:36:23.0943515Z activate-with-label: false 2025-10-10T00:36:23.0943729Z label: with-ssh 2025-10-10T00:36:23.0943922Z remove-existing-keys: true 2025-10-10T00:36:23.0944129Z fail-silently: true 2025-10-10T00:36:23.0944322Z env: 2025-10-10T00:36:23.0944493Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:36:23.0944707Z ##[endgroup] 2025-10-10T00:36:23.1967444Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-10-10T00:36:23.1968848Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-10-10T00:36:23.2101182Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-10-10T00:36:23.2101571Z with: 2025-10-10T00:36:23.2101786Z no-sudo: true 2025-10-10T00:36:23.2102014Z submodules: recursive 2025-10-10T00:36:23.2102253Z fetch-depth: 0 2025-10-10T00:36:23.2102468Z env: 2025-10-10T00:36:23.2102673Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:36:23.2102945Z ##[endgroup] 2025-10-10T00:36:23.2185703Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:36:23.2186359Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:36:23.2195090Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:36:23.2195347Z env: 2025-10-10T00:36:23.2195556Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:36:23.2195733Z ##[endgroup] 2025-10-10T00:36:23.2280636Z ##[group]Run # Use all available CPUs for fetching 2025-10-10T00:36:23.2281049Z # Use all available CPUs for fetching 2025-10-10T00:36:23.2281334Z cd "${GITHUB_WORKSPACE}" 2025-10-10T00:36:23.2281642Z git config --global fetch.parallel 0 2025-10-10T00:36:23.2281928Z git config --global submodule.fetchJobs 0 2025-10-10T00:36:23.2282170Z  2025-10-10T00:36:23.2282451Z # Clean workspace. The default checkout action should also do this, but 2025-10-10T00:36:23.2282776Z # do it here as well just in case 2025-10-10T00:36:23.2283009Z if [[ -d .git ]]; then 2025-10-10T00:36:23.2283231Z  if [ -z "${NO_SUDO}" ]; then 2025-10-10T00:36:23.2283460Z  sudo git clean -ffdx 2025-10-10T00:36:23.2283662Z  else 2025-10-10T00:36:23.2283845Z  git clean -ffdx 2025-10-10T00:36:23.2284041Z  fi 2025-10-10T00:36:23.2284209Z fi 2025-10-10T00:36:23.2288896Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:36:23.2289169Z env: 2025-10-10T00:36:23.2289465Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:36:23.2289690Z NO_SUDO: true 2025-10-10T00:36:23.2289871Z ##[endgroup] 2025-10-10T00:36:23.2404793Z ##[group]Run actions/checkout@v4 2025-10-10T00:36:23.2405064Z with: 2025-10-10T00:36:23.2405289Z ref: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:36:23.2405733Z fetch-depth: 0 2025-10-10T00:36:23.2405944Z submodules: recursive 2025-10-10T00:36:23.2406158Z show-progress: false 2025-10-10T00:36:23.2406372Z repository: pytorch/pytorch 2025-10-10T00:36:23.2406687Z token: *** 2025-10-10T00:36:23.2406863Z ssh-strict: true 2025-10-10T00:36:23.2407048Z ssh-user: git 2025-10-10T00:36:23.2407239Z persist-credentials: true 2025-10-10T00:36:23.2407446Z clean: true 2025-10-10T00:36:23.2407644Z sparse-checkout-cone-mode: true 2025-10-10T00:36:23.2407869Z fetch-tags: false 2025-10-10T00:36:23.2408052Z lfs: false 2025-10-10T00:36:23.2408236Z set-safe-directory: true 2025-10-10T00:36:23.2408453Z env: 2025-10-10T00:36:23.2408634Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:36:23.2408840Z ##[endgroup] 2025-10-10T00:36:23.3374547Z Syncing repository: pytorch/pytorch 2025-10-10T00:36:23.3375711Z ##[group]Getting Git version info 2025-10-10T00:36:23.3376054Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-10-10T00:36:23.3376558Z [command]/usr/bin/git version 2025-10-10T00:36:23.3376793Z git version 2.50.1 2025-10-10T00:36:23.3399014Z ##[endgroup] 2025-10-10T00:36:23.3402649Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/03f0a743-ed69-4c94-8bf8-10885ca35a00/.gitconfig' 2025-10-10T00:36:23.3441170Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/03f0a743-ed69-4c94-8bf8-10885ca35a00' before making global git config changes 2025-10-10T00:36:23.3446679Z Adding repository directory to the temporary git global config as a safe directory 2025-10-10T00:36:23.3454446Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T00:36:23.3493785Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-10-10T00:36:23.3496336Z ##[group]Initializing the repository 2025-10-10T00:36:23.3496691Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T00:36:23.3525951Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-10-10T00:36:23.3526334Z hint: is subject to change. To configure the initial branch name to use in all 2025-10-10T00:36:23.3527444Z hint: of your new repositories, which will suppress this warning, call: 2025-10-10T00:36:23.3527784Z hint: 2025-10-10T00:36:23.3528016Z hint: git config --global init.defaultBranch 2025-10-10T00:36:23.3528250Z hint: 2025-10-10T00:36:23.3528495Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-10-10T00:36:23.3528880Z hint: 'development'. The just-created branch can be renamed via this command: 2025-10-10T00:36:23.3529179Z hint: 2025-10-10T00:36:23.3529354Z hint: git branch -m 2025-10-10T00:36:23.3529542Z hint: 2025-10-10T00:36:23.3529796Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-10-10T00:36:23.3530207Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-10-10T00:36:23.3538303Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-10-10T00:36:23.3574033Z ##[endgroup] 2025-10-10T00:36:23.3574436Z ##[group]Disabling automatic garbage collection 2025-10-10T00:36:23.3574735Z [command]/usr/bin/git config --local gc.auto 0 2025-10-10T00:36:23.3612194Z ##[endgroup] 2025-10-10T00:36:23.3612590Z ##[group]Setting up auth 2025-10-10T00:36:23.3612979Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-10-10T00:36:23.3646780Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-10-10T00:36:23.3961580Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-10-10T00:36:23.3992621Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-10-10T00:36:23.4294795Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-10-10T00:36:23.4350714Z ##[endgroup] 2025-10-10T00:36:23.4356449Z ##[group]Fetching the repository 2025-10-10T00:36:23.4358466Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-10-10T00:37:12.8961841Z From https://github.com/pytorch/pytorch 2025-10-10T00:37:12.8965367Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-10-10T00:37:12.8975328Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-10-10T00:37:12.8976991Z * [new branch] BootcampDynamo -> origin/BootcampDynamo 2025-10-10T00:37:12.8977449Z * [new branch] DynamoFixGit -> origin/DynamoFixGit 2025-10-10T00:37:12.8980744Z * [new branch] DynamoVariaT -> origin/DynamoVariaT 2025-10-10T00:37:12.8981150Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-10-10T00:37:12.8981540Z * [new branch] ISSUE-154849 -> origin/ISSUE-154849 2025-10-10T00:37:12.8981890Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-10-10T00:37:12.8982268Z * [new branch] IvanKobzarev/stack/2 -> origin/IvanKobzarev/stack/2 2025-10-10T00:37:12.8982960Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-10-10T00:37:12.8983334Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-10-10T00:37:12.8983751Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-10-10T00:37:12.8984147Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-10-10T00:37:12.8984555Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-10-10T00:37:12.8984922Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-10-10T00:37:12.8985295Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-10-10T00:37:12.8985659Z * [new branch] VLA_exp -> origin/VLA_exp 2025-10-10T00:37:12.8986056Z * [new branch] actually-run-mps-aot-inductor -> origin/actually-run-mps-aot-inductor 2025-10-10T00:37:12.8986695Z * [new branch] add_compile_benchmarking -> origin/add_compile_benchmarking 2025-10-10T00:37:12.8987103Z * [new branch] add_op_tests -> origin/add_op_tests 2025-10-10T00:37:12.8987434Z * [new branch] add_op_to_dashboard -> origin/add_op_to_dashboard 2025-10-10T00:37:12.8987779Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-10-10T00:37:12.8988113Z * [new branch] addremovefunction -> origin/addremovefunction 2025-10-10T00:37:12.8988435Z * [new branch] addvllmtest -> origin/addvllmtest 2025-10-10T00:37:12.8988753Z * [new branch] adi/test -> origin/adi/test 2025-10-10T00:37:12.8989073Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-10-10T00:37:12.8989427Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-10-10T00:37:12.8989777Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-10-10T00:37:12.8990215Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-10-10T00:37:12.8990575Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-10-10T00:37:12.8990921Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-10-10T00:37:12.8991395Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-10-10T00:37:12.8991772Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-10-10T00:37:12.8992166Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-10-10T00:37:12.8992539Z * [new branch] alt-disable -> origin/alt-disable 2025-10-10T00:37:12.8992936Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-10-10T00:37:12.8993351Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-10-10T00:37:12.8993719Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-10-10T00:37:12.8994087Z * [new branch] angelayi/benchmark3 -> origin/angelayi/benchmark3 2025-10-10T00:37:12.8994516Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-10-10T00:37:12.8994959Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-10-10T00:37:12.8995312Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-10-10T00:37:12.8995764Z * [new branch] angelayi/fix_mps -> origin/angelayi/fix_mps 2025-10-10T00:37:12.8996123Z * [new branch] angelayi/lint -> origin/angelayi/lint 2025-10-10T00:37:12.8996441Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-10-10T00:37:12.8996860Z * [new branch] angelayi/opaque_obj_v2 -> origin/angelayi/opaque_obj_v2 2025-10-10T00:37:12.8997205Z * [new branch] angelayi/pattern -> origin/angelayi/pattern 2025-10-10T00:37:12.8997558Z * [new branch] angelayi/pattern_in_out_2 -> origin/angelayi/pattern_in_out_2 2025-10-10T00:37:12.8997919Z * [new branch] angelayi/post_grad -> origin/angelayi/post_grad 2025-10-10T00:37:12.8998262Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-10-10T00:37:12.8998625Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-10-10T00:37:12.8998993Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-10-10T00:37:12.8999362Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-10-10T00:37:12.8999716Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-10-10T00:37:12.9000065Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-10-10T00:37:12.9000428Z * [new branch] angelayi/wrap_grad -> origin/angelayi/wrap_grad 2025-10-10T00:37:12.9000766Z * [new branch] annotate_1 -> origin/annotate_1 2025-10-10T00:37:12.9001138Z * [new branch] annotation_bw -> origin/annotation_bw 2025-10-10T00:37:12.9001478Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-10-10T00:37:12.9001812Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-10-10T00:37:12.9002156Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-10-10T00:37:12.9002505Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-10-10T00:37:12.9004873Z * [new branch] aoti_metal_shimify -> origin/aoti_metal_shimify 2025-10-10T00:37:12.9005255Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-10-10T00:37:12.9005689Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-10-10T00:37:12.9006031Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-10-10T00:37:12.9006426Z * [new branch] aoti_windows_mingw -> origin/aoti_windows_mingw 2025-10-10T00:37:12.9006768Z * [new branch] aoti_windows_mingw_2 -> origin/aoti_windows_mingw_2 2025-10-10T00:37:12.9007174Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-10-10T00:37:12.9007569Z * [new branch] async_tp -> origin/async_tp 2025-10-10T00:37:12.9007921Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-10-10T00:37:12.9008351Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-10-10T00:37:12.9010045Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-10-10T00:37:12.9010412Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-10-10T00:37:12.9010760Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-10-10T00:37:12.9011119Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-10-10T00:37:12.9011445Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-10-10T00:37:12.9011776Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-10-10T00:37:12.9013800Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-10-10T00:37:12.9014158Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-10-10T00:37:12.9014576Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-10-10T00:37:12.9014927Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-10-10T00:37:12.9015291Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-10-10T00:37:12.9015657Z * [new branch] attention_benchmark -> origin/attention_benchmark 2025-10-10T00:37:12.9018747Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-10-10T00:37:12.9019123Z * [new branch] b200_op_bench -> origin/b200_op_bench 2025-10-10T00:37:12.9019449Z * [new branch] bahuang/annotation -> origin/bahuang/annotation 2025-10-10T00:37:12.9019799Z * [new branch] bahuang/debug_mode -> origin/bahuang/debug_mode 2025-10-10T00:37:12.9020167Z * [new branch] bahuang/debug_mode_default -> origin/bahuang/debug_mode_default 2025-10-10T00:37:12.9020722Z * [new branch] bahuang/debug_mode_fix -> origin/bahuang/debug_mode_fix 2025-10-10T00:37:12.9021241Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-10-10T00:37:12.9027142Z * [new branch] bahuang/dt_reduce_mean -> origin/bahuang/dt_reduce_mean 2025-10-10T00:37:12.9030562Z * [new branch] bahuang/dtensor_demo -> origin/bahuang/dtensor_demo 2025-10-10T00:37:12.9030972Z * [new branch] bahuang/export_dtensor -> origin/bahuang/export_dtensor 2025-10-10T00:37:12.9031391Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-10-10T00:37:12.9032030Z * [new branch] bahuang/fix_debug_mode2 -> origin/bahuang/fix_debug_mode2 2025-10-10T00:37:12.9032402Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-10-10T00:37:12.9032812Z * [new branch] bahuang/noop_redistribute -> origin/bahuang/noop_redistribute 2025-10-10T00:37:12.9033187Z * [new branch] bahuang/reland -> origin/bahuang/reland 2025-10-10T00:37:12.9033555Z * [new branch] bahuang/reland_fake_export -> origin/bahuang/reland_fake_export 2025-10-10T00:37:12.9033908Z * [new branch] bahuang/rename -> origin/bahuang/rename 2025-10-10T00:37:12.9034482Z * [new branch] bahuang/test -> origin/bahuang/test 2025-10-10T00:37:12.9034803Z * [new branch] base/1.5 -> origin/base/1.5 2025-10-10T00:37:12.9035189Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-10-10T00:37:12.9035639Z * [new branch] bc-lint-test-new-config -> origin/bc-lint-test-new-config 2025-10-10T00:37:12.9036011Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-10-10T00:37:12.9036382Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-10-10T00:37:12.9036757Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-10-10T00:37:12.9037676Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-10-10T00:37:12.9038040Z * [new branch] bf/cg-custom-wrapper -> origin/bf/cg-custom-wrapper 2025-10-10T00:37:12.9038406Z * [new branch] bf/cg-error-re-record -> origin/bf/cg-error-re-record 2025-10-10T00:37:12.9038850Z * [new branch] bf/cg-partition-custom-op-mutation -> origin/bf/cg-partition-custom-op-mutation 2025-10-10T00:37:12.9041553Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-10-10T00:37:12.9042028Z * [new branch] bf/cg-warn-dynamic-shapes -> origin/bf/cg-warn-dynamic-shapes 2025-10-10T00:37:12.9048040Z * [new branch] bf/cherry-pick-partition-share-default-device-context -> origin/bf/cherry-pick-partition-share-default-device-context 2025-10-10T00:37:12.9048680Z * [new branch] bf/clean-hf -> origin/bf/clean-hf 2025-10-10T00:37:12.9049044Z * [new branch] bf/clean-timm -> origin/bf/clean-timm 2025-10-10T00:37:12.9049400Z * [new branch] bf/clean-torchbench -> origin/bf/clean-torchbench 2025-10-10T00:37:12.9049802Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-10-10T00:37:12.9050166Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-10-10T00:37:12.9050601Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-10-10T00:37:12.9051203Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-10-10T00:37:12.9051748Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-10-10T00:37:12.9052132Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-10-10T00:37:12.9053035Z * [new branch] bf/minor-cg-config-doc -> origin/bf/minor-cg-config-doc 2025-10-10T00:37:12.9053412Z * [new branch] bf/minor-fa-tma-config -> origin/bf/minor-fa-tma-config 2025-10-10T00:37:12.9053792Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-10-10T00:37:12.9054196Z * [new branch] bf/partition-custom-op-alias -> origin/bf/partition-custom-op-alias 2025-10-10T00:37:12.9054660Z * [new branch] bf/partition-default-device-context -> origin/bf/partition-default-device-context 2025-10-10T00:37:12.9055083Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-10-10T00:37:12.9055457Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-10-10T00:37:12.9055813Z * [new branch] bf/rope -> origin/bf/rope 2025-10-10T00:37:12.9057066Z * [new branch] bf16_support -> origin/bf16_support 2025-10-10T00:37:12.9057433Z * [new branch] bf16_support_per_channel -> origin/bf16_support_per_channel 2025-10-10T00:37:12.9057822Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-10-10T00:37:12.9058312Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-10-10T00:37:12.9058708Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-10-10T00:37:12.9059094Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-10-10T00:37:12.9059473Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-10-10T00:37:12.9062739Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-10-10T00:37:12.9063175Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-10-10T00:37:12.9063610Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-10-10T00:37:12.9064013Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-10-10T00:37:12.9064444Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-10-10T00:37:12.9064837Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-10-10T00:37:12.9065247Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-10-10T00:37:12.9065652Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-10-10T00:37:12.9066121Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-10-10T00:37:12.9066674Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-10-10T00:37:12.9067074Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-10-10T00:37:12.9067459Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-10-10T00:37:12.9067869Z * [new branch] brister/break_scatter_src_is_tensor -> origin/brister/break_scatter_src_is_tensor 2025-10-10T00:37:12.9068269Z * [new branch] brister/fx_cond -> origin/brister/fx_cond 2025-10-10T00:37:12.9068621Z * [new branch] brister/fx_dynamic_input -> origin/brister/fx_dynamic_input 2025-10-10T00:37:12.9068984Z * [new branch] brister/fx_index_put -> origin/brister/fx_index_put 2025-10-10T00:37:12.9074225Z * [new branch] brister/fx_no_python_slow -> origin/brister/fx_no_python_slow 2025-10-10T00:37:12.9074699Z * [new branch] brister/fx_scatter_reduce -> origin/brister/fx_scatter_reduce 2025-10-10T00:37:12.9075120Z * [new branch] brister/fx_unbacked_symbols -> origin/brister/fx_unbacked_symbols 2025-10-10T00:37:12.9075549Z * [new branch] brister/property_type_check -> origin/brister/property_type_check 2025-10-10T00:37:12.9075973Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-10-10T00:37:12.9076451Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-10-10T00:37:12.9076898Z * [new branch] build-aarch64-wheels -> origin/build-aarch64-wheels 2025-10-10T00:37:12.9077261Z * [new branch] bwd-backup -> origin/bwd-backup 2025-10-10T00:37:12.9077590Z * [new branch] c57382a49 -> origin/c57382a49 2025-10-10T00:37:12.9077918Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-10-10T00:37:12.9078310Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-10-10T00:37:12.9079171Z * [new branch] camyll/cherrypick_0098e5636d3afa7c75aef8c447a5c402ea9ed524 -> origin/camyll/cherrypick_0098e5636d3afa7c75aef8c447a5c402ea9ed524 2025-10-10T00:37:12.9080505Z * [new branch] camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 -> origin/camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 2025-10-10T00:37:12.9081367Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-10-10T00:37:12.9082208Z * [new branch] camyll/revert_5d749ceb92c2c28bcfbdf918b4ab99b1a91fcb50 -> origin/camyll/revert_5d749ceb92c2c28bcfbdf918b4ab99b1a91fcb50 2025-10-10T00:37:12.9083001Z * [new branch] camyllh/cherrypick_5e7be988003a38be49227cfaa9bff6a2ea9e6929_v2 -> origin/camyllh/cherrypick_5e7be988003a38be49227cfaa9bff6a2ea9e6929_v2 2025-10-10T00:37:12.9083764Z * [new branch] camyllh/cherrypick_dda071587f0522a16b237f92cbe27fd13a1a1c11 -> origin/camyllh/cherrypick_dda071587f0522a16b237f92cbe27fd13a1a1c11 2025-10-10T00:37:12.9084573Z * [new branch] camyllh/release2_9_cherrypick/dda071587f0522a16b237f92cbe27fd13a1a1c11 -> origin/camyllh/release2_9_cherrypick/dda071587f0522a16b237f92cbe27fd13a1a1c11 2025-10-10T00:37:12.9085225Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-10-10T00:37:12.9088723Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9089346Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9090084Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9090608Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9091134Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9091653Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9092242Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9092752Z * [new branch] cherry-pick-161299-by-pytorch_bot_bot_ -> origin/cherry-pick-161299-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9093312Z * [new branch] cherry-pick-161394-by-pytorch_bot_bot_ -> origin/cherry-pick-161394-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9093825Z * [new branch] cherry-pick-161430-by-pytorch_bot_bot_ -> origin/cherry-pick-161430-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9094345Z * [new branch] cherry-pick-162168-by-pytorch_bot_bot_ -> origin/cherry-pick-162168-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9094856Z * [new branch] cherry-pick-162194-by-pytorch_bot_bot_ -> origin/cherry-pick-162194-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9095380Z * [new branch] cherry-pick-162240-by-pytorch_bot_bot_ -> origin/cherry-pick-162240-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9095899Z * [new branch] cherry-pick-162295-by-pytorch_bot_bot_ -> origin/cherry-pick-162295-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9096483Z * [new branch] cherry-pick-162323-by-pytorch_bot_bot_ -> origin/cherry-pick-162323-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9097025Z * [new branch] cherry-pick-162425-by-pytorch_bot_bot_ -> origin/cherry-pick-162425-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9097807Z * [new branch] cherry-pick-162530-by-pytorch_bot_bot_ -> origin/cherry-pick-162530-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9098336Z * [new branch] cherry-pick-162555-by-pytorch_bot_bot_ -> origin/cherry-pick-162555-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9098925Z * [new branch] cherry-pick-162566-by-pytorch_bot_bot_ -> origin/cherry-pick-162566-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9099452Z * [new branch] cherry-pick-162587-by-pytorch_bot_bot_ -> origin/cherry-pick-162587-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9100092Z * [new branch] cherry-pick-162622-by-pytorch_bot_bot_ -> origin/cherry-pick-162622-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9100945Z * [new branch] cherry-pick-162657-by-pytorch_bot_bot_ -> origin/cherry-pick-162657-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9101722Z * [new branch] cherry-pick-162680-by-pytorch_bot_bot_ -> origin/cherry-pick-162680-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9102645Z * [new branch] cherry-pick-162693-by-pytorch_bot_bot_ -> origin/cherry-pick-162693-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9103521Z * [new branch] cherry-pick-162744-by-pytorch_bot_bot_ -> origin/cherry-pick-162744-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9105093Z * [new branch] cherry-pick-162764-by-pytorch_bot_bot_ -> origin/cherry-pick-162764-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9105706Z * [new branch] cherry-pick-162865-by-pytorch_bot_bot_ -> origin/cherry-pick-162865-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9106243Z * [new branch] cherry-pick-162866-by-pytorch_bot_bot_ -> origin/cherry-pick-162866-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9106971Z * [new branch] cherry-pick-162877-by-pytorch_bot_bot_ -> origin/cherry-pick-162877-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9107865Z * [new branch] cherry-pick-162950-by-pytorch_bot_bot_ -> origin/cherry-pick-162950-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9108806Z * [new branch] cherry-pick-163008-by-pytorch_bot_bot_ -> origin/cherry-pick-163008-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9110484Z * [new branch] cherry-pick-163111-by-pytorch_bot_bot_ -> origin/cherry-pick-163111-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9111018Z * [new branch] cherry-pick-163112-by-pytorch_bot_bot_ -> origin/cherry-pick-163112-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9111549Z * [new branch] cherry-pick-163152-by-pytorch_bot_bot_ -> origin/cherry-pick-163152-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9112062Z * [new branch] cherry-pick-163171-by-pytorch_bot_bot_ -> origin/cherry-pick-163171-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9112564Z * [new branch] cherry-pick-163194-by-pytorch_bot_bot_ -> origin/cherry-pick-163194-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9114215Z * [new branch] cherry-pick-163227-by-pytorch_bot_bot_ -> origin/cherry-pick-163227-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9114779Z * [new branch] cherry-pick-163269-by-pytorch_bot_bot_ -> origin/cherry-pick-163269-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9115604Z * [new branch] cherry-pick-163298-by-pytorch_bot_bot_ -> origin/cherry-pick-163298-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9117490Z * [new branch] cherry-pick-163315-by-pytorch_bot_bot_ -> origin/cherry-pick-163315-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9118109Z * [new branch] cherry-pick-163339-by-pytorch_bot_bot_ -> origin/cherry-pick-163339-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9118656Z * [new branch] cherry-pick-163341-by-pytorch_bot_bot_ -> origin/cherry-pick-163341-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9119242Z * [new branch] cherry-pick-163370-by-pytorch_bot_bot_ -> origin/cherry-pick-163370-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9119739Z * [new branch] cherry-pick-163383-by-pytorch_bot_bot_ -> origin/cherry-pick-163383-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9120786Z * [new branch] cherry-pick-163426-by-pytorch_bot_bot_ -> origin/cherry-pick-163426-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9121386Z * [new branch] cherry-pick-163549-by-pytorch_bot_bot_ -> origin/cherry-pick-163549-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9121923Z * [new branch] cherry-pick-163571-by-pytorch_bot_bot_ -> origin/cherry-pick-163571-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9122658Z * [new branch] cherry-pick-163578-by-pytorch_bot_bot_ -> origin/cherry-pick-163578-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9123379Z * [new branch] cherry-pick-163581-by-pytorch_bot_bot_ -> origin/cherry-pick-163581-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9123899Z * [new branch] cherry-pick-163585-by-pytorch_bot_bot_ -> origin/cherry-pick-163585-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9124739Z * [new branch] cherry-pick-163587-by-pytorch_bot_bot_ -> origin/cherry-pick-163587-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9125536Z * [new branch] cherry-pick-163598-by-pytorch_bot_bot_ -> origin/cherry-pick-163598-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9126317Z * [new branch] cherry-pick-163661-by-pytorch_bot_bot_ -> origin/cherry-pick-163661-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9127149Z * [new branch] cherry-pick-163677-by-pytorch_bot_bot_ -> origin/cherry-pick-163677-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9127927Z * [new branch] cherry-pick-163682-by-pytorch_bot_bot_ -> origin/cherry-pick-163682-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9128601Z * [new branch] cherry-pick-163712-by-pytorch_bot_bot_ -> origin/cherry-pick-163712-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9129406Z * [new branch] cherry-pick-163719-by-pytorch_bot_bot_ -> origin/cherry-pick-163719-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9130354Z * [new branch] cherry-pick-163768-by-pytorch_bot_bot_ -> origin/cherry-pick-163768-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9131302Z * [new branch] cherry-pick-163776-by-pytorch_bot_bot_ -> origin/cherry-pick-163776-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9132266Z * [new branch] cherry-pick-163797-by-pytorch_bot_bot_ -> origin/cherry-pick-163797-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9133075Z * [new branch] cherry-pick-163837-by-pytorch_bot_bot_ -> origin/cherry-pick-163837-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9133824Z * [new branch] cherry-pick-163886-by-pytorch_bot_bot_ -> origin/cherry-pick-163886-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9134591Z * [new branch] cherry-pick-163903-by-pytorch_bot_bot_ -> origin/cherry-pick-163903-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9135492Z * [new branch] cherry-pick-163956-by-pytorch_bot_bot_ -> origin/cherry-pick-163956-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9136414Z * [new branch] cherry-pick-163988-by-pytorch_bot_bot_ -> origin/cherry-pick-163988-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9137208Z * [new branch] cherry-pick-164093-by-pytorch_bot_bot_ -> origin/cherry-pick-164093-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9137996Z * [new branch] cherry-pick-164108-by-pytorch_bot_bot_ -> origin/cherry-pick-164108-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9138887Z * [new branch] cherry-pick-164138-by-pytorch_bot_bot_ -> origin/cherry-pick-164138-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9139552Z * [new branch] cherry-pick-164190 -> origin/cherry-pick-164190 2025-10-10T00:37:12.9140382Z * [new branch] cherry-pick-164470-by-pytorch_bot_bot_ -> origin/cherry-pick-164470-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9141347Z * [new branch] cherry-pick-164575-by-pytorch_bot_bot_ -> origin/cherry-pick-164575-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9142169Z * [new branch] cherry-pick-164774-by-pytorch_bot_bot_ -> origin/cherry-pick-164774-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9143155Z * [new branch] cherry-pick-164870-by-pytorch_bot_bot_ -> origin/cherry-pick-164870-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9143975Z * [new branch] cherry-pick-164946-by-pytorch_bot_bot_ -> origin/cherry-pick-164946-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9144709Z * [new branch] cherry-pick-165013-by-pytorch_bot_bot_ -> origin/cherry-pick-165013-by-pytorch_bot_bot_ 2025-10-10T00:37:12.9145499Z * [new branch] cherry_pick_graph_custom -> origin/cherry_pick_graph_custom 2025-10-10T00:37:12.9147216Z * [new branch] cherrypick-e88cca0691 -> origin/cherrypick-e88cca0691 2025-10-10T00:37:12.9147626Z * [new branch] chuanqi129-patch-1 -> origin/chuanqi129-patch-1 2025-10-10T00:37:12.9147983Z * [new branch] ck_dlpack -> origin/ck_dlpack 2025-10-10T00:37:12.9148674Z * [new branch] codegen_trace -> origin/codegen_trace 2025-10-10T00:37:12.9149664Z * [new branch] codex-testing -> origin/codex-testing 2025-10-10T00:37:12.9151833Z * [new branch] codex/add-metadata-field-for-file-path -> origin/codex/add-metadata-field-for-file-path 2025-10-10T00:37:12.9152485Z * [new branch] codex/add-test-for-inductor-local-cache-behavior -> origin/codex/add-test-for-inductor-local-cache-behavior 2025-10-10T00:37:12.9153208Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-10-10T00:37:12.9154453Z * [new branch] codex/enhance-cuda.matmul-with-allow_splitk-argument -> origin/codex/enhance-cuda.matmul-with-allow_splitk-argument 2025-10-10T00:37:12.9155048Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-10-10T00:37:12.9155620Z * [new branch] codex/refactor-dimension-handling-in-shape.cu -> origin/codex/refactor-dimension-handling-in-shape.cu 2025-10-10T00:37:12.9156555Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-10-10T00:37:12.9157259Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors 2025-10-10T00:37:12.9158039Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors-vx0jek -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors-vx0jek 2025-10-10T00:37:12.9158708Z * [new branch] compile_kernel_include_dir -> origin/compile_kernel_include_dir 2025-10-10T00:37:12.9159083Z * [new branch] context_test -> origin/context_test 2025-10-10T00:37:12.9159441Z * [new branch] conv1d_decomp -> origin/conv1d_decomp 2025-10-10T00:37:12.9160445Z * [new branch] conv_autotune -> origin/conv_autotune 2025-10-10T00:37:12.9161631Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-10-10T00:37:12.9162609Z * [new branch] copilot/fix-163730 -> origin/copilot/fix-163730 2025-10-10T00:37:12.9163080Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-10-10T00:37:12.9163983Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-10-10T00:37:12.9164678Z * [new branch] cpp_head -> origin/cpp_head 2025-10-10T00:37:12.9166757Z * [new branch] crcrpar-patch-1 -> origin/crcrpar-patch-1 2025-10-10T00:37:12.9167187Z * [new branch] csl/add_win_shard -> origin/csl/add_win_shard 2025-10-10T00:37:12.9167582Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-10-10T00:37:12.9168164Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-10-10T00:37:12.9168585Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-10-10T00:37:12.9169491Z * [new branch] csl/fix_internal_graph_executor -> origin/csl/fix_internal_graph_executor 2025-10-10T00:37:12.9170303Z * [new branch] csl/fix_nightly_docs_push -> origin/csl/fix_nightly_docs_push 2025-10-10T00:37:12.9171480Z * [new branch] csl/inductor_h100_nightly -> origin/csl/inductor_h100_nightly 2025-10-10T00:37:12.9172061Z * [new branch] csl/katex -> origin/csl/katex 2025-10-10T00:37:12.9173168Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-10-10T00:37:12.9173649Z * [new branch] csl/lint_no_submodules -> origin/csl/lint_no_submodules 2025-10-10T00:37:12.9174083Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-10-10T00:37:12.9174726Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-10-10T00:37:12.9176828Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-10-10T00:37:12.9177287Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-10-10T00:37:12.9177662Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-10-10T00:37:12.9178240Z * [new branch] csl/reuse_old_whl_fix_metadata -> origin/csl/reuse_old_whl_fix_metadata 2025-10-10T00:37:12.9178768Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-10-10T00:37:12.9179343Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-10-10T00:37:12.9180211Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-10-10T00:37:12.9180863Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-10-10T00:37:12.9181571Z * [new branch] csl/test_info_status -> origin/csl/test_info_status 2025-10-10T00:37:12.9182497Z * [new branch] csl/test_info_upload_changes -> origin/csl/test_info_upload_changes 2025-10-10T00:37:12.9182916Z * [new branch] csl/test_owners_ao_sparse -> origin/csl/test_owners_ao_sparse 2025-10-10T00:37:12.9184156Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-10-10T00:37:12.9184601Z * [new branch] csl/test_owners_cuda -> origin/csl/test_owners_cuda 2025-10-10T00:37:12.9185346Z * [new branch] csl/test_owners_distributed -> origin/csl/test_owners_distributed 2025-10-10T00:37:12.9186161Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-10-10T00:37:12.9186825Z * [new branch] csl/testing_better_job_name -> origin/csl/testing_better_job_name 2025-10-10T00:37:12.9187690Z * [new branch] csl/vllm_pin_labeler -> origin/csl/vllm_pin_labeler 2025-10-10T00:37:12.9188380Z * [new branch] csl/win_cpp_tests -> origin/csl/win_cpp_tests 2025-10-10T00:37:12.9189032Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-10-10T00:37:12.9189935Z * [new branch] cu_stream_api -> origin/cu_stream_api 2025-10-10T00:37:12.9190676Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-10-10T00:37:12.9191453Z * [new branch] cublasnowdeterministic -> origin/cublasnowdeterministic 2025-10-10T00:37:12.9192210Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-10-10T00:37:12.9192985Z * [new branch] cuda-include-paths-fix -> origin/cuda-include-paths-fix 2025-10-10T00:37:12.9193723Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-10-10T00:37:12.9198897Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-10-10T00:37:12.9199332Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-10-10T00:37:12.9199673Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-10-10T00:37:12.9200005Z * [new branch] debug-guard -> origin/debug-guard 2025-10-10T00:37:12.9200360Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-10-10T00:37:12.9201157Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-10-10T00:37:12.9207028Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-10-10T00:37:12.9209587Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-10-10T00:37:12.9210060Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-10-10T00:37:12.9210482Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-10-10T00:37:12.9210869Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-10-10T00:37:12.9211207Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-10-10T00:37:12.9211564Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-10-10T00:37:12.9211973Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-10-10T00:37:12.9212443Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-10-10T00:37:12.9213810Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-10-10T00:37:12.9214193Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-10-10T00:37:12.9214553Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-10-10T00:37:12.9215057Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-10-10T00:37:12.9217567Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-10-10T00:37:12.9217914Z * [new branch] disable -> origin/disable 2025-10-10T00:37:12.9218226Z * [new branch] disp_counter -> origin/disp_counter 2025-10-10T00:37:12.9221857Z * [new branch] dtensor-issues -> origin/dtensor-issues 2025-10-10T00:37:12.9222759Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-10-10T00:37:12.9223218Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-10-10T00:37:12.9223613Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-10-10T00:37:12.9224085Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-10-10T00:37:12.9224519Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-10-10T00:37:12.9224930Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-10-10T00:37:12.9225384Z * [new branch] enable-keep-going-for-trunk-tags -> origin/enable-keep-going-for-trunk-tags 2025-10-10T00:37:12.9225827Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-10-10T00:37:12.9226156Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-10-10T00:37:12.9230191Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-10-10T00:37:12.9230718Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-10-10T00:37:12.9231158Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-10-10T00:37:12.9231771Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-10-10T00:37:12.9232288Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-10-10T00:37:12.9236675Z * [new branch] exclamaforte/fix-exhuastive-autotuning-reland -> origin/exclamaforte/fix-exhuastive-autotuning-reland 2025-10-10T00:37:12.9240259Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-10-10T00:37:12.9242391Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-10-10T00:37:12.9243048Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-10-10T00:37:12.9249119Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-10-10T00:37:12.9254498Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-10-10T00:37:12.9256769Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-10-10T00:37:12.9257382Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-10-10T00:37:12.9261927Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-10-10T00:37:12.9262458Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-10-10T00:37:12.9262974Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-10-10T00:37:12.9263512Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-10-10T00:37:12.9264006Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-10-10T00:37:12.9264722Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-10-10T00:37:12.9265236Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-10-10T00:37:12.9265783Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-10-10T00:37:12.9266569Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-10-10T00:37:12.9266965Z * [new branch] exec -> origin/exec 2025-10-10T00:37:12.9267310Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-10-10T00:37:12.9267687Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-10-10T00:37:12.9268060Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-10-10T00:37:12.9268399Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-10-10T00:37:12.9268720Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-10-10T00:37:12.9269034Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-10-10T00:37:12.9269354Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-10-10T00:37:12.9269680Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-10-10T00:37:12.9270001Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-10-10T00:37:12.9270320Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-10-10T00:37:12.9270633Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-10-10T00:37:12.9270955Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-10-10T00:37:12.9271278Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-10-10T00:37:12.9271602Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-10-10T00:37:12.9271925Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-10-10T00:37:12.9272243Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-10-10T00:37:12.9272646Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-10-10T00:37:12.9272970Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-10-10T00:37:12.9273296Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-10-10T00:37:12.9273612Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-10-10T00:37:12.9273935Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-10-10T00:37:12.9274264Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-10-10T00:37:12.9274569Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-10-10T00:37:12.9274873Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-10-10T00:37:12.9275169Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-10-10T00:37:12.9275497Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-10-10T00:37:12.9275798Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-10-10T00:37:12.9276100Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-10-10T00:37:12.9276409Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-10-10T00:37:12.9276705Z * [new branch] export-D79378362 -> origin/export-D79378362 2025-10-10T00:37:12.9277046Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-10-10T00:37:12.9277355Z * [new branch] export-D80948073 -> origin/export-D80948073 2025-10-10T00:37:12.9277660Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-10-10T00:37:12.9277961Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-10-10T00:37:12.9278261Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-10-10T00:37:12.9278564Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-10-10T00:37:12.9278868Z * [new branch] export-D81651226 -> origin/export-D81651226 2025-10-10T00:37:12.9279647Z * [new branch] export-D81698719 -> origin/export-D81698719 2025-10-10T00:37:12.9279966Z * [new branch] export-D82140619 -> origin/export-D82140619 2025-10-10T00:37:12.9280289Z * [new branch] export-D82174075 -> origin/export-D82174075 2025-10-10T00:37:12.9280622Z * [new branch] export-D82232574 -> origin/export-D82232574 2025-10-10T00:37:12.9280930Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-10-10T00:37:12.9281327Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-10-10T00:37:12.9286266Z * [new branch] export-D82380307 -> origin/export-D82380307 2025-10-10T00:37:12.9290384Z * [new branch] export-D82597111 -> origin/export-D82597111 2025-10-10T00:37:12.9292211Z * [new branch] export-D83023706 -> origin/export-D83023706 2025-10-10T00:37:12.9292576Z * [new branch] export-D83195687 -> origin/export-D83195687 2025-10-10T00:37:12.9292924Z * [new branch] export-D83200714 -> origin/export-D83200714 2025-10-10T00:37:12.9293280Z * [new branch] export-D83378477 -> origin/export-D83378477 2025-10-10T00:37:12.9293629Z * [new branch] export-D83390563 -> origin/export-D83390563 2025-10-10T00:37:12.9293971Z * [new branch] export-D83390775 -> origin/export-D83390775 2025-10-10T00:37:12.9294328Z * [new branch] export-D83391942 -> origin/export-D83391942 2025-10-10T00:37:12.9294739Z * [new branch] export-D83395610 -> origin/export-D83395610 2025-10-10T00:37:12.9295083Z * [new branch] export-D83539263 -> origin/export-D83539263 2025-10-10T00:37:12.9295424Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-10-10T00:37:12.9295757Z * [new branch] export-D83591083 -> origin/export-D83591083 2025-10-10T00:37:12.9296133Z * [new branch] export-D83609850 -> origin/export-D83609850 2025-10-10T00:37:12.9296477Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-10-10T00:37:12.9296817Z * [new branch] export-D83714690 -> origin/export-D83714690 2025-10-10T00:37:12.9297154Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-10-10T00:37:12.9297512Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-10-10T00:37:12.9297873Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-10-10T00:37:12.9298216Z * [new branch] export-D84009392 -> origin/export-D84009392 2025-10-10T00:37:12.9298562Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-10-10T00:37:12.9298929Z * [new branch] export-D84098898 -> origin/export-D84098898 2025-10-10T00:37:12.9299274Z * [new branch] export-D84103213 -> origin/export-D84103213 2025-10-10T00:37:12.9299667Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-10-10T00:37:12.9300020Z * [new branch] export-reland -> origin/export-reland 2025-10-10T00:37:12.9300453Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-10-10T00:37:12.9300925Z * [new branch] extend_lift_up_op -> origin/extend_lift_up_op 2025-10-10T00:37:12.9303664Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-10-10T00:37:12.9304050Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-10-10T00:37:12.9304395Z * [new branch] ezyang-war -> origin/ezyang-war 2025-10-10T00:37:12.9304766Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-10-10T00:37:12.9305142Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-10-10T00:37:12.9305603Z * [new branch] fadeputr-fix-fbgemm_genai-build -> origin/fadeputr-fix-fbgemm_genai-build 2025-10-10T00:37:12.9306041Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-10-10T00:37:12.9308060Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-10-10T00:37:12.9308412Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-10-10T00:37:12.9308747Z * [new branch] fca -> origin/fca 2025-10-10T00:37:12.9309052Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-10-10T00:37:12.9311099Z * [new branch] fca5 -> origin/fca5 2025-10-10T00:37:12.9311440Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-10-10T00:37:12.9311805Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-10-10T00:37:12.9314280Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-10-10T00:37:12.9314694Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-10-10T00:37:12.9315057Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-10-10T00:37:12.9315423Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-10-10T00:37:12.9317882Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-10-10T00:37:12.9318253Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-10-10T00:37:12.9318630Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-10-10T00:37:12.9319015Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-10-10T00:37:12.9319386Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-10-10T00:37:12.9319789Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-10-10T00:37:12.9320171Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-10-10T00:37:12.9322807Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-10-10T00:37:12.9323138Z * [new branch] fix-fqn -> origin/fix-fqn 2025-10-10T00:37:12.9323509Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-10-10T00:37:12.9323951Z * [new branch] fix-upload-vllm-wheel-credential -> origin/fix-upload-vllm-wheel-credential 2025-10-10T00:37:12.9326327Z * [new branch] fix_153389 -> origin/fix_153389 2025-10-10T00:37:12.9326672Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-10-10T00:37:12.9327045Z * [new branch] fix_op_benchmark -> origin/fix_op_benchmark 2025-10-10T00:37:12.9327494Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-10-10T00:37:12.9327826Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-10-10T00:37:12.9328483Z * [new branch] fixes -> origin/fixes 2025-10-10T00:37:12.9328854Z * [new branch] fixes-triage -> origin/fixes-triage 2025-10-10T00:37:12.9329234Z * [new branch] fixflashgit -> origin/fixflashgit 2025-10-10T00:37:12.9330351Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-10-10T00:37:12.9330731Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-10-10T00:37:12.9331765Z * [new branch] flex-flash -> origin/flex-flash 2025-10-10T00:37:12.9332577Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-10-10T00:37:12.9333309Z * [new branch] flex_flash -> origin/flex_flash 2025-10-10T00:37:12.9336695Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-10-10T00:37:12.9337342Z * [new branch] free-stack2 -> origin/free-stack2 2025-10-10T00:37:12.9337835Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-10-10T00:37:12.9338209Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-10-10T00:37:12.9338552Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-10-10T00:37:12.9338990Z * [new branch] fused_moving_avg_obs_fake_quant_half_support -> origin/fused_moving_avg_obs_fake_quant_half_support 2025-10-10T00:37:12.9339448Z * [new branch] fx_cpp -> origin/fx_cpp 2025-10-10T00:37:12.9341428Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-10-10T00:37:12.9342659Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-10-10T00:37:12.9343139Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-10-10T00:37:12.9344936Z * [new branch] gh/ColinPeppler/80/base -> origin/gh/ColinPeppler/80/base 2025-10-10T00:37:12.9345723Z * [new branch] gh/ColinPeppler/80/head -> origin/gh/ColinPeppler/80/head 2025-10-10T00:37:12.9346733Z * [new branch] gh/ColinPeppler/80/orig -> origin/gh/ColinPeppler/80/orig 2025-10-10T00:37:12.9347642Z * [new branch] gh/ColinPeppler/81/base -> origin/gh/ColinPeppler/81/base 2025-10-10T00:37:12.9348326Z * [new branch] gh/ColinPeppler/81/head -> origin/gh/ColinPeppler/81/head 2025-10-10T00:37:12.9349108Z * [new branch] gh/ColinPeppler/81/orig -> origin/gh/ColinPeppler/81/orig 2025-10-10T00:37:12.9349984Z * [new branch] gh/ColinPeppler/82/base -> origin/gh/ColinPeppler/82/base 2025-10-10T00:37:12.9350721Z * [new branch] gh/ColinPeppler/82/head -> origin/gh/ColinPeppler/82/head 2025-10-10T00:37:12.9351511Z * [new branch] gh/ColinPeppler/82/orig -> origin/gh/ColinPeppler/82/orig 2025-10-10T00:37:12.9352850Z * [new branch] gh/ColinPeppler/83/base -> origin/gh/ColinPeppler/83/base 2025-10-10T00:37:12.9353512Z * [new branch] gh/ColinPeppler/83/head -> origin/gh/ColinPeppler/83/head 2025-10-10T00:37:12.9354328Z * [new branch] gh/ColinPeppler/83/orig -> origin/gh/ColinPeppler/83/orig 2025-10-10T00:37:12.9355743Z * [new branch] gh/ColinPeppler/84/base -> origin/gh/ColinPeppler/84/base 2025-10-10T00:37:12.9356387Z * [new branch] gh/ColinPeppler/84/head -> origin/gh/ColinPeppler/84/head 2025-10-10T00:37:12.9357019Z * [new branch] gh/ColinPeppler/85/base -> origin/gh/ColinPeppler/85/base 2025-10-10T00:37:12.9357970Z * [new branch] gh/ColinPeppler/85/head -> origin/gh/ColinPeppler/85/head 2025-10-10T00:37:12.9358594Z * [new branch] gh/ColinPeppler/86/base -> origin/gh/ColinPeppler/86/base 2025-10-10T00:37:12.9359248Z * [new branch] gh/ColinPeppler/86/head -> origin/gh/ColinPeppler/86/head 2025-10-10T00:37:12.9360474Z * [new branch] gh/ColinPeppler/87/base -> origin/gh/ColinPeppler/87/base 2025-10-10T00:37:12.9360900Z * [new branch] gh/ColinPeppler/87/head -> origin/gh/ColinPeppler/87/head 2025-10-10T00:37:12.9362553Z * [new branch] gh/ColinPeppler/88/base -> origin/gh/ColinPeppler/88/base 2025-10-10T00:37:12.9363006Z * [new branch] gh/ColinPeppler/88/head -> origin/gh/ColinPeppler/88/head 2025-10-10T00:37:12.9363640Z * [new branch] gh/ColinPeppler/89/base -> origin/gh/ColinPeppler/89/base 2025-10-10T00:37:12.9364300Z * [new branch] gh/ColinPeppler/89/head -> origin/gh/ColinPeppler/89/head 2025-10-10T00:37:12.9365178Z * [new branch] gh/ColinPeppler/90/base -> origin/gh/ColinPeppler/90/base 2025-10-10T00:37:12.9365830Z * [new branch] gh/ColinPeppler/90/head -> origin/gh/ColinPeppler/90/head 2025-10-10T00:37:12.9366686Z * [new branch] gh/ColinPeppler/91/base -> origin/gh/ColinPeppler/91/base 2025-10-10T00:37:12.9367452Z * [new branch] gh/ColinPeppler/91/head -> origin/gh/ColinPeppler/91/head 2025-10-10T00:37:12.9368418Z * [new branch] gh/ColinPeppler/92/base -> origin/gh/ColinPeppler/92/base 2025-10-10T00:37:12.9369047Z * [new branch] gh/ColinPeppler/92/head -> origin/gh/ColinPeppler/92/head 2025-10-10T00:37:12.9370626Z * [new branch] gh/ColinPeppler/93/base -> origin/gh/ColinPeppler/93/base 2025-10-10T00:37:12.9371022Z * [new branch] gh/ColinPeppler/93/head -> origin/gh/ColinPeppler/93/head 2025-10-10T00:37:12.9371828Z * [new branch] gh/ColinPeppler/93/orig -> origin/gh/ColinPeppler/93/orig 2025-10-10T00:37:12.9373372Z * [new branch] gh/ColinPeppler/94/base -> origin/gh/ColinPeppler/94/base 2025-10-10T00:37:12.9374105Z * [new branch] gh/ColinPeppler/94/head -> origin/gh/ColinPeppler/94/head 2025-10-10T00:37:12.9374813Z * [new branch] gh/ColinPeppler/94/orig -> origin/gh/ColinPeppler/94/orig 2025-10-10T00:37:12.9376428Z * [new branch] gh/ColinPeppler/95/base -> origin/gh/ColinPeppler/95/base 2025-10-10T00:37:12.9376983Z * [new branch] gh/ColinPeppler/95/head -> origin/gh/ColinPeppler/95/head 2025-10-10T00:37:12.9377677Z * [new branch] gh/ColinPeppler/95/orig -> origin/gh/ColinPeppler/95/orig 2025-10-10T00:37:12.9379266Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-10-10T00:37:12.9379667Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-10-10T00:37:12.9381592Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-10-10T00:37:12.9381996Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-10-10T00:37:12.9383646Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-10-10T00:37:12.9384270Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-10-10T00:37:12.9384913Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-10-10T00:37:12.9386223Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-10-10T00:37:12.9386730Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-10-10T00:37:12.9387406Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-10-10T00:37:12.9388798Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-10-10T00:37:12.9389260Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-10-10T00:37:12.9389894Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-10-10T00:37:12.9390972Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-10-10T00:37:12.9391351Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-10-10T00:37:12.9394354Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-10-10T00:37:12.9394705Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-10-10T00:37:12.9395053Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-10-10T00:37:12.9395559Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-10-10T00:37:12.9395923Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-10-10T00:37:12.9396263Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-10-10T00:37:12.9396624Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-10-10T00:37:12.9399856Z * [new branch] gh/H-Huang/210/base -> origin/gh/H-Huang/210/base 2025-10-10T00:37:12.9400213Z * [new branch] gh/H-Huang/210/head -> origin/gh/H-Huang/210/head 2025-10-10T00:37:12.9400547Z * [new branch] gh/H-Huang/210/orig -> origin/gh/H-Huang/210/orig 2025-10-10T00:37:12.9400888Z * [new branch] gh/H-Huang/212/base -> origin/gh/H-Huang/212/base 2025-10-10T00:37:12.9401227Z * [new branch] gh/H-Huang/212/head -> origin/gh/H-Huang/212/head 2025-10-10T00:37:12.9403828Z * [new branch] gh/H-Huang/212/orig -> origin/gh/H-Huang/212/orig 2025-10-10T00:37:12.9404173Z * [new branch] gh/H-Huang/214/base -> origin/gh/H-Huang/214/base 2025-10-10T00:37:12.9404512Z * [new branch] gh/H-Huang/214/head -> origin/gh/H-Huang/214/head 2025-10-10T00:37:12.9404849Z * [new branch] gh/H-Huang/214/orig -> origin/gh/H-Huang/214/orig 2025-10-10T00:37:12.9407483Z * [new branch] gh/H-Huang/215/base -> origin/gh/H-Huang/215/base 2025-10-10T00:37:12.9407873Z * [new branch] gh/H-Huang/215/head -> origin/gh/H-Huang/215/head 2025-10-10T00:37:12.9408202Z * [new branch] gh/H-Huang/215/orig -> origin/gh/H-Huang/215/orig 2025-10-10T00:37:12.9408543Z * [new branch] gh/H-Huang/216/base -> origin/gh/H-Huang/216/base 2025-10-10T00:37:12.9408884Z * [new branch] gh/H-Huang/216/head -> origin/gh/H-Huang/216/head 2025-10-10T00:37:12.9411779Z * [new branch] gh/H-Huang/216/orig -> origin/gh/H-Huang/216/orig 2025-10-10T00:37:12.9412135Z * [new branch] gh/H-Huang/217/base -> origin/gh/H-Huang/217/base 2025-10-10T00:37:12.9412475Z * [new branch] gh/H-Huang/217/head -> origin/gh/H-Huang/217/head 2025-10-10T00:37:12.9412806Z * [new branch] gh/H-Huang/217/orig -> origin/gh/H-Huang/217/orig 2025-10-10T00:37:12.9413188Z * [new branch] gh/H-Huang/218/base -> origin/gh/H-Huang/218/base 2025-10-10T00:37:12.9413529Z * [new branch] gh/H-Huang/218/head -> origin/gh/H-Huang/218/head 2025-10-10T00:37:12.9415695Z * [new branch] gh/H-Huang/218/orig -> origin/gh/H-Huang/218/orig 2025-10-10T00:37:12.9416042Z * [new branch] gh/H-Huang/219/base -> origin/gh/H-Huang/219/base 2025-10-10T00:37:12.9416372Z * [new branch] gh/H-Huang/219/head -> origin/gh/H-Huang/219/head 2025-10-10T00:37:12.9416712Z * [new branch] gh/H-Huang/219/orig -> origin/gh/H-Huang/219/orig 2025-10-10T00:37:12.9417179Z * [new branch] gh/H-Huang/220/base -> origin/gh/H-Huang/220/base 2025-10-10T00:37:12.9419355Z * [new branch] gh/H-Huang/220/head -> origin/gh/H-Huang/220/head 2025-10-10T00:37:12.9419706Z * [new branch] gh/H-Huang/220/orig -> origin/gh/H-Huang/220/orig 2025-10-10T00:37:12.9420035Z * [new branch] gh/H-Huang/221/base -> origin/gh/H-Huang/221/base 2025-10-10T00:37:12.9420462Z * [new branch] gh/H-Huang/221/head -> origin/gh/H-Huang/221/head 2025-10-10T00:37:12.9420808Z * [new branch] gh/H-Huang/221/orig -> origin/gh/H-Huang/221/orig 2025-10-10T00:37:12.9422932Z * [new branch] gh/H-Huang/222/base -> origin/gh/H-Huang/222/base 2025-10-10T00:37:12.9423279Z * [new branch] gh/H-Huang/222/head -> origin/gh/H-Huang/222/head 2025-10-10T00:37:12.9423619Z * [new branch] gh/H-Huang/222/orig -> origin/gh/H-Huang/222/orig 2025-10-10T00:37:12.9423962Z * [new branch] gh/H-Huang/223/base -> origin/gh/H-Huang/223/base 2025-10-10T00:37:12.9424317Z * [new branch] gh/H-Huang/223/head -> origin/gh/H-Huang/223/head 2025-10-10T00:37:12.9424978Z * [new branch] gh/H-Huang/223/orig -> origin/gh/H-Huang/223/orig 2025-10-10T00:37:12.9426544Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-10-10T00:37:12.9430586Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-10-10T00:37:12.9431018Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-10-10T00:37:12.9431416Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-10-10T00:37:12.9431996Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-10-10T00:37:12.9432391Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-10-10T00:37:12.9435094Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-10-10T00:37:12.9435546Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-10-10T00:37:12.9441359Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-10-10T00:37:12.9446066Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-10-10T00:37:12.9450975Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-10-10T00:37:12.9455673Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-10-10T00:37:12.9457844Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-10-10T00:37:12.9458464Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-10-10T00:37:12.9458993Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-10-10T00:37:12.9459488Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-10-10T00:37:12.9460012Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-10-10T00:37:12.9460512Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-10-10T00:37:12.9460879Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-10-10T00:37:12.9461250Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-10-10T00:37:12.9461605Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-10-10T00:37:12.9461957Z * [new branch] gh/IvanKobzarev/138/base -> origin/gh/IvanKobzarev/138/base 2025-10-10T00:37:12.9462482Z * [new branch] gh/IvanKobzarev/138/head -> origin/gh/IvanKobzarev/138/head 2025-10-10T00:37:12.9462850Z * [new branch] gh/IvanKobzarev/138/orig -> origin/gh/IvanKobzarev/138/orig 2025-10-10T00:37:12.9463223Z * [new branch] gh/IvanKobzarev/141/base -> origin/gh/IvanKobzarev/141/base 2025-10-10T00:37:12.9463594Z * [new branch] gh/IvanKobzarev/141/head -> origin/gh/IvanKobzarev/141/head 2025-10-10T00:37:12.9463974Z * [new branch] gh/IvanKobzarev/141/orig -> origin/gh/IvanKobzarev/141/orig 2025-10-10T00:37:12.9464346Z * [new branch] gh/IvanKobzarev/142/base -> origin/gh/IvanKobzarev/142/base 2025-10-10T00:37:12.9464712Z * [new branch] gh/IvanKobzarev/142/head -> origin/gh/IvanKobzarev/142/head 2025-10-10T00:37:12.9465086Z * [new branch] gh/IvanKobzarev/142/orig -> origin/gh/IvanKobzarev/142/orig 2025-10-10T00:37:12.9465457Z * [new branch] gh/IvanKobzarev/144/base -> origin/gh/IvanKobzarev/144/base 2025-10-10T00:37:12.9465828Z * [new branch] gh/IvanKobzarev/144/head -> origin/gh/IvanKobzarev/144/head 2025-10-10T00:37:12.9466190Z * [new branch] gh/IvanKobzarev/144/orig -> origin/gh/IvanKobzarev/144/orig 2025-10-10T00:37:12.9466761Z * [new branch] gh/IvanKobzarev/145/base -> origin/gh/IvanKobzarev/145/base 2025-10-10T00:37:12.9467140Z * [new branch] gh/IvanKobzarev/145/head -> origin/gh/IvanKobzarev/145/head 2025-10-10T00:37:12.9467514Z * [new branch] gh/IvanKobzarev/145/orig -> origin/gh/IvanKobzarev/145/orig 2025-10-10T00:37:12.9467893Z * [new branch] gh/IvanKobzarev/146/base -> origin/gh/IvanKobzarev/146/base 2025-10-10T00:37:12.9468227Z * [new branch] gh/IvanKobzarev/146/head -> origin/gh/IvanKobzarev/146/head 2025-10-10T00:37:12.9468571Z * [new branch] gh/IvanKobzarev/146/orig -> origin/gh/IvanKobzarev/146/orig 2025-10-10T00:37:12.9468918Z * [new branch] gh/IvanKobzarev/147/base -> origin/gh/IvanKobzarev/147/base 2025-10-10T00:37:12.9469284Z * [new branch] gh/IvanKobzarev/147/head -> origin/gh/IvanKobzarev/147/head 2025-10-10T00:37:12.9469663Z * [new branch] gh/IvanKobzarev/147/orig -> origin/gh/IvanKobzarev/147/orig 2025-10-10T00:37:12.9470030Z * [new branch] gh/IvanKobzarev/148/base -> origin/gh/IvanKobzarev/148/base 2025-10-10T00:37:12.9470431Z * [new branch] gh/IvanKobzarev/148/head -> origin/gh/IvanKobzarev/148/head 2025-10-10T00:37:12.9470772Z * [new branch] gh/IvanKobzarev/149/base -> origin/gh/IvanKobzarev/149/base 2025-10-10T00:37:12.9471135Z * [new branch] gh/IvanKobzarev/149/head -> origin/gh/IvanKobzarev/149/head 2025-10-10T00:37:12.9471505Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-10-10T00:37:12.9471883Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-10-10T00:37:12.9476462Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-10-10T00:37:12.9478811Z * [new branch] gh/IvanKobzarev/151/base -> origin/gh/IvanKobzarev/151/base 2025-10-10T00:37:12.9479286Z * [new branch] gh/IvanKobzarev/151/head -> origin/gh/IvanKobzarev/151/head 2025-10-10T00:37:12.9479696Z * [new branch] gh/IvanKobzarev/151/orig -> origin/gh/IvanKobzarev/151/orig 2025-10-10T00:37:12.9485546Z * [new branch] gh/IvanKobzarev/152/base -> origin/gh/IvanKobzarev/152/base 2025-10-10T00:37:12.9490444Z * [new branch] gh/IvanKobzarev/152/head -> origin/gh/IvanKobzarev/152/head 2025-10-10T00:37:12.9494257Z * [new branch] gh/IvanKobzarev/152/orig -> origin/gh/IvanKobzarev/152/orig 2025-10-10T00:37:12.9498610Z * [new branch] gh/IvanKobzarev/153/base -> origin/gh/IvanKobzarev/153/base 2025-10-10T00:37:12.9504379Z * [new branch] gh/IvanKobzarev/153/head -> origin/gh/IvanKobzarev/153/head 2025-10-10T00:37:12.9509369Z * [new branch] gh/IvanKobzarev/153/orig -> origin/gh/IvanKobzarev/153/orig 2025-10-10T00:37:12.9509851Z * [new branch] gh/IvanKobzarev/154/base -> origin/gh/IvanKobzarev/154/base 2025-10-10T00:37:12.9510244Z * [new branch] gh/IvanKobzarev/154/head -> origin/gh/IvanKobzarev/154/head 2025-10-10T00:37:12.9510659Z * [new branch] gh/IvanKobzarev/154/orig -> origin/gh/IvanKobzarev/154/orig 2025-10-10T00:37:12.9511044Z * [new branch] gh/IvanKobzarev/155/base -> origin/gh/IvanKobzarev/155/base 2025-10-10T00:37:12.9511419Z * [new branch] gh/IvanKobzarev/155/head -> origin/gh/IvanKobzarev/155/head 2025-10-10T00:37:12.9511794Z * [new branch] gh/IvanKobzarev/155/orig -> origin/gh/IvanKobzarev/155/orig 2025-10-10T00:37:12.9512149Z * [new branch] gh/IvanKobzarev/156/base -> origin/gh/IvanKobzarev/156/base 2025-10-10T00:37:12.9512503Z * [new branch] gh/IvanKobzarev/156/head -> origin/gh/IvanKobzarev/156/head 2025-10-10T00:37:12.9512856Z * [new branch] gh/IvanKobzarev/156/orig -> origin/gh/IvanKobzarev/156/orig 2025-10-10T00:37:12.9513207Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-10-10T00:37:12.9513564Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-10-10T00:37:12.9513908Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-10-10T00:37:12.9514259Z * [new branch] gh/IvanKobzarev/158/base -> origin/gh/IvanKobzarev/158/base 2025-10-10T00:37:12.9514607Z * [new branch] gh/IvanKobzarev/158/head -> origin/gh/IvanKobzarev/158/head 2025-10-10T00:37:12.9514969Z * [new branch] gh/IvanKobzarev/158/orig -> origin/gh/IvanKobzarev/158/orig 2025-10-10T00:37:12.9515346Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-10-10T00:37:12.9515709Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-10-10T00:37:12.9516081Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-10-10T00:37:12.9516451Z * [new branch] gh/IvanKobzarev/160/base -> origin/gh/IvanKobzarev/160/base 2025-10-10T00:37:12.9516954Z * [new branch] gh/IvanKobzarev/160/head -> origin/gh/IvanKobzarev/160/head 2025-10-10T00:37:12.9517308Z * [new branch] gh/IvanKobzarev/160/orig -> origin/gh/IvanKobzarev/160/orig 2025-10-10T00:37:12.9517722Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-10-10T00:37:12.9518120Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-10-10T00:37:12.9518500Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-10-10T00:37:12.9518873Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-10-10T00:37:12.9519242Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-10-10T00:37:12.9519694Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-10-10T00:37:12.9520071Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-10-10T00:37:12.9520409Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-10-10T00:37:12.9520733Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-10-10T00:37:12.9521065Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-10-10T00:37:12.9521398Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-10-10T00:37:12.9521781Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-10-10T00:37:12.9522111Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-10-10T00:37:12.9522437Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-10-10T00:37:12.9522761Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-10-10T00:37:12.9523086Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-10-10T00:37:12.9523419Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-10-10T00:37:12.9523752Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-10-10T00:37:12.9524084Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-10-10T00:37:12.9524411Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-10-10T00:37:12.9524745Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-10-10T00:37:12.9525071Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-10-10T00:37:12.9525399Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-10-10T00:37:12.9525722Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-10-10T00:37:12.9526043Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-10-10T00:37:12.9526386Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-10-10T00:37:12.9526708Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-10-10T00:37:12.9527055Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-10-10T00:37:12.9527411Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-10-10T00:37:12.9527744Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-10-10T00:37:12.9528058Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-10-10T00:37:12.9528369Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-10-10T00:37:12.9528697Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-10-10T00:37:12.9529080Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-10-10T00:37:12.9529392Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-10-10T00:37:12.9529712Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-10-10T00:37:12.9530027Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-10-10T00:37:12.9530348Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-10-10T00:37:12.9530666Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-10-10T00:37:12.9530976Z * [new branch] gh/PaliC/27/head -> origin/gh/PaliC/27/head 2025-10-10T00:37:12.9531292Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-10-10T00:37:12.9531775Z * [new branch] gh/PaliC/27/orig -> origin/gh/PaliC/27/orig 2025-10-10T00:37:12.9532092Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-10-10T00:37:12.9532404Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-10-10T00:37:12.9532723Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-10-10T00:37:12.9533069Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-10-10T00:37:12.9533405Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-10-10T00:37:12.9533834Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-10-10T00:37:12.9534151Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-10-10T00:37:12.9534459Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-10-10T00:37:12.9534768Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-10-10T00:37:12.9535081Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-10-10T00:37:12.9535396Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-10-10T00:37:12.9535714Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-10-10T00:37:12.9536052Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-10-10T00:37:12.9536409Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-10-10T00:37:12.9536761Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-10-10T00:37:12.9537939Z * [new branch] gh/PaulZhang12/24/base -> origin/gh/PaulZhang12/24/base 2025-10-10T00:37:12.9538384Z * [new branch] gh/PaulZhang12/24/head -> origin/gh/PaulZhang12/24/head 2025-10-10T00:37:12.9542714Z * [new branch] gh/PaulZhang12/24/orig -> origin/gh/PaulZhang12/24/orig 2025-10-10T00:37:12.9543319Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-10-10T00:37:12.9543815Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-10-10T00:37:12.9544341Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-10-10T00:37:12.9544825Z * [new branch] gh/PaulZhang12/26/base -> origin/gh/PaulZhang12/26/base 2025-10-10T00:37:12.9545319Z * [new branch] gh/PaulZhang12/26/head -> origin/gh/PaulZhang12/26/head 2025-10-10T00:37:12.9545830Z * [new branch] gh/PaulZhang12/26/orig -> origin/gh/PaulZhang12/26/orig 2025-10-10T00:37:12.9546409Z * [new branch] gh/PaulZhang12/27/base -> origin/gh/PaulZhang12/27/base 2025-10-10T00:37:12.9547299Z * [new branch] gh/PaulZhang12/27/head -> origin/gh/PaulZhang12/27/head 2025-10-10T00:37:12.9547714Z * [new branch] gh/PaulZhang12/27/orig -> origin/gh/PaulZhang12/27/orig 2025-10-10T00:37:12.9548296Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-10-10T00:37:12.9548679Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-10-10T00:37:12.9549054Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-10-10T00:37:12.9549418Z * [new branch] gh/PaulZhang12/29/base -> origin/gh/PaulZhang12/29/base 2025-10-10T00:37:12.9553692Z * [new branch] gh/PaulZhang12/29/head -> origin/gh/PaulZhang12/29/head 2025-10-10T00:37:12.9554289Z * [new branch] gh/PaulZhang12/29/orig -> origin/gh/PaulZhang12/29/orig 2025-10-10T00:37:12.9554823Z * [new branch] gh/PaulZhang12/30/base -> origin/gh/PaulZhang12/30/base 2025-10-10T00:37:12.9555209Z * [new branch] gh/PaulZhang12/30/head -> origin/gh/PaulZhang12/30/head 2025-10-10T00:37:12.9555582Z * [new branch] gh/PaulZhang12/30/orig -> origin/gh/PaulZhang12/30/orig 2025-10-10T00:37:12.9555950Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-10-10T00:37:12.9556319Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-10-10T00:37:12.9560948Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-10-10T00:37:12.9561347Z * [new branch] gh/PaulZhang12/32/base -> origin/gh/PaulZhang12/32/base 2025-10-10T00:37:12.9561926Z * [new branch] gh/PaulZhang12/32/head -> origin/gh/PaulZhang12/32/head 2025-10-10T00:37:12.9562269Z * [new branch] gh/PaulZhang12/32/orig -> origin/gh/PaulZhang12/32/orig 2025-10-10T00:37:12.9562613Z * [new branch] gh/PaulZhang12/33/base -> origin/gh/PaulZhang12/33/base 2025-10-10T00:37:12.9562949Z * [new branch] gh/PaulZhang12/33/head -> origin/gh/PaulZhang12/33/head 2025-10-10T00:37:12.9563294Z * [new branch] gh/PaulZhang12/33/orig -> origin/gh/PaulZhang12/33/orig 2025-10-10T00:37:12.9563667Z * [new branch] gh/PaulZhang12/34/base -> origin/gh/PaulZhang12/34/base 2025-10-10T00:37:12.9563999Z * [new branch] gh/PaulZhang12/34/head -> origin/gh/PaulZhang12/34/head 2025-10-10T00:37:12.9564329Z * [new branch] gh/PaulZhang12/34/orig -> origin/gh/PaulZhang12/34/orig 2025-10-10T00:37:12.9564662Z * [new branch] gh/PaulZhang12/35/base -> origin/gh/PaulZhang12/35/base 2025-10-10T00:37:12.9565005Z * [new branch] gh/PaulZhang12/35/head -> origin/gh/PaulZhang12/35/head 2025-10-10T00:37:12.9565352Z * [new branch] gh/PaulZhang12/35/orig -> origin/gh/PaulZhang12/35/orig 2025-10-10T00:37:12.9568381Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-10-10T00:37:12.9568952Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-10-10T00:37:12.9569532Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-10-10T00:37:12.9570037Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-10-10T00:37:12.9570959Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-10-10T00:37:12.9571553Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-10-10T00:37:12.9572067Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-10-10T00:37:12.9572891Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-10-10T00:37:12.9573355Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-10-10T00:37:12.9573760Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-10-10T00:37:12.9574280Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-10-10T00:37:12.9574702Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-10-10T00:37:12.9575072Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-10-10T00:37:12.9575431Z * [new branch] gh/SherlockNoMad/13/base -> origin/gh/SherlockNoMad/13/base 2025-10-10T00:37:12.9575966Z * [new branch] gh/SherlockNoMad/13/head -> origin/gh/SherlockNoMad/13/head 2025-10-10T00:37:12.9576587Z * [new branch] gh/SherlockNoMad/13/orig -> origin/gh/SherlockNoMad/13/orig 2025-10-10T00:37:12.9580227Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-10-10T00:37:12.9580864Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-10-10T00:37:12.9581368Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-10-10T00:37:12.9581739Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-10-10T00:37:12.9582109Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-10-10T00:37:12.9582466Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-10-10T00:37:12.9582818Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-10-10T00:37:12.9583310Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-10-10T00:37:12.9583668Z * [new branch] gh/SherlockNoMad/6/base -> origin/gh/SherlockNoMad/6/base 2025-10-10T00:37:12.9584169Z * [new branch] gh/SherlockNoMad/6/head -> origin/gh/SherlockNoMad/6/head 2025-10-10T00:37:12.9584533Z * [new branch] gh/SherlockNoMad/6/orig -> origin/gh/SherlockNoMad/6/orig 2025-10-10T00:37:12.9584899Z * [new branch] gh/SherlockNoMad/7/base -> origin/gh/SherlockNoMad/7/base 2025-10-10T00:37:12.9585257Z * [new branch] gh/SherlockNoMad/7/head -> origin/gh/SherlockNoMad/7/head 2025-10-10T00:37:12.9585739Z * [new branch] gh/SherlockNoMad/7/orig -> origin/gh/SherlockNoMad/7/orig 2025-10-10T00:37:12.9586677Z * [new branch] gh/SherlockNoMad/8/base -> origin/gh/SherlockNoMad/8/base 2025-10-10T00:37:12.9587352Z * [new branch] gh/SherlockNoMad/8/head -> origin/gh/SherlockNoMad/8/head 2025-10-10T00:37:12.9588058Z * [new branch] gh/SherlockNoMad/8/orig -> origin/gh/SherlockNoMad/8/orig 2025-10-10T00:37:12.9593281Z * [new branch] gh/SherlockNoMad/9/base -> origin/gh/SherlockNoMad/9/base 2025-10-10T00:37:12.9593854Z * [new branch] gh/SherlockNoMad/9/orig -> origin/gh/SherlockNoMad/9/orig 2025-10-10T00:37:12.9594367Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-10-10T00:37:12.9594777Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-10-10T00:37:12.9595155Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-10-10T00:37:12.9605927Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-10-10T00:37:12.9610926Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-10-10T00:37:12.9615957Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-10-10T00:37:12.9616412Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-10-10T00:37:12.9616761Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-10-10T00:37:12.9617111Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-10-10T00:37:12.9617599Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-10-10T00:37:12.9617984Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-10-10T00:37:12.9618366Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-10-10T00:37:12.9618741Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-10-10T00:37:12.9619126Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-10-10T00:37:12.9619507Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-10-10T00:37:12.9619849Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-10-10T00:37:12.9620191Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-10-10T00:37:12.9620526Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-10-10T00:37:12.9620874Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-10-10T00:37:12.9621220Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-10-10T00:37:12.9621590Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-10-10T00:37:12.9621964Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-10-10T00:37:12.9622331Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-10-10T00:37:12.9622754Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-10-10T00:37:12.9623131Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-10-10T00:37:12.9623498Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-10-10T00:37:12.9623845Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-10-10T00:37:12.9624205Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-10-10T00:37:12.9624568Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-10-10T00:37:12.9624932Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-10-10T00:37:12.9625286Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-10-10T00:37:12.9625634Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-10-10T00:37:12.9625995Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-10-10T00:37:12.9626533Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-10-10T00:37:12.9626897Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-10-10T00:37:12.9627268Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-10-10T00:37:12.9627624Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-10-10T00:37:12.9627973Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-10-10T00:37:12.9628329Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-10-10T00:37:12.9628689Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-10-10T00:37:12.9629049Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-10-10T00:37:12.9629403Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-10-10T00:37:12.9629762Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-10-10T00:37:12.9630115Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-10-10T00:37:12.9630516Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-10-10T00:37:12.9630868Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-10-10T00:37:12.9631219Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-10-10T00:37:12.9631712Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-10-10T00:37:12.9632073Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-10-10T00:37:12.9632589Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-10-10T00:37:12.9639028Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-10-10T00:37:12.9644096Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-10-10T00:37:12.9648598Z * [new branch] gh/XilunWu/166/base -> origin/gh/XilunWu/166/base 2025-10-10T00:37:12.9650488Z * [new branch] gh/XilunWu/166/head -> origin/gh/XilunWu/166/head 2025-10-10T00:37:12.9650834Z * [new branch] gh/XilunWu/166/orig -> origin/gh/XilunWu/166/orig 2025-10-10T00:37:12.9651176Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-10-10T00:37:12.9651496Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-10-10T00:37:12.9651808Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-10-10T00:37:12.9652311Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-10-10T00:37:12.9652633Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-10-10T00:37:12.9652951Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-10-10T00:37:12.9653265Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-10-10T00:37:12.9653579Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-10-10T00:37:12.9653894Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-10-10T00:37:12.9654212Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-10-10T00:37:12.9654533Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-10-10T00:37:12.9654858Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-10-10T00:37:12.9655168Z * [new branch] gh/XilunWu/172/base -> origin/gh/XilunWu/172/base 2025-10-10T00:37:12.9655482Z * [new branch] gh/XilunWu/172/head -> origin/gh/XilunWu/172/head 2025-10-10T00:37:12.9655795Z * [new branch] gh/XilunWu/172/orig -> origin/gh/XilunWu/172/orig 2025-10-10T00:37:12.9656109Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-10-10T00:37:12.9656426Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-10-10T00:37:12.9656735Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-10-10T00:37:12.9657056Z * [new branch] gh/XilunWu/174/base -> origin/gh/XilunWu/174/base 2025-10-10T00:37:12.9657374Z * [new branch] gh/XilunWu/174/head -> origin/gh/XilunWu/174/head 2025-10-10T00:37:12.9657691Z * [new branch] gh/XilunWu/174/orig -> origin/gh/XilunWu/174/orig 2025-10-10T00:37:12.9658019Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-10-10T00:37:12.9658747Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-10-10T00:37:12.9659083Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-10-10T00:37:12.9659416Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-10-10T00:37:12.9659900Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-10-10T00:37:12.9665347Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-10-10T00:37:12.9670498Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-10-10T00:37:12.9675943Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-10-10T00:37:12.9681579Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-10-10T00:37:12.9683510Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-10-10T00:37:12.9684006Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-10-10T00:37:12.9688385Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-10-10T00:37:12.9688798Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-10-10T00:37:12.9691867Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-10-10T00:37:12.9692221Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-10-10T00:37:12.9692555Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-10-10T00:37:12.9692887Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-10-10T00:37:12.9693440Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-10-10T00:37:12.9693784Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-10-10T00:37:12.9694112Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-10-10T00:37:12.9694476Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-10-10T00:37:12.9694815Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-10-10T00:37:12.9695151Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-10-10T00:37:12.9695484Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-10-10T00:37:12.9695810Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-10-10T00:37:12.9696138Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-10-10T00:37:12.9696476Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-10-10T00:37:12.9696811Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-10-10T00:37:12.9697149Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-10-10T00:37:12.9697478Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-10-10T00:37:12.9697818Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-10-10T00:37:12.9698155Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-10-10T00:37:12.9698473Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-10-10T00:37:12.9698785Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-10-10T00:37:12.9699108Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-10-10T00:37:12.9699433Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-10-10T00:37:12.9699750Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-10-10T00:37:12.9700075Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-10-10T00:37:12.9700449Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-10-10T00:37:12.9700767Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-10-10T00:37:12.9701090Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-10-10T00:37:12.9701416Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-10-10T00:37:12.9701743Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-10-10T00:37:12.9702067Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-10-10T00:37:12.9702394Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-10-10T00:37:12.9702728Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-10-10T00:37:12.9703065Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-10-10T00:37:12.9703405Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-10-10T00:37:12.9703734Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-10-10T00:37:12.9704071Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-10-10T00:37:12.9704407Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-10-10T00:37:12.9704749Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-10-10T00:37:12.9705318Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-10-10T00:37:12.9705682Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-10-10T00:37:12.9706045Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-10-10T00:37:12.9706611Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-10-10T00:37:12.9706959Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-10-10T00:37:12.9707296Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-10-10T00:37:12.9707624Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-10-10T00:37:12.9707972Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-10-10T00:37:12.9708302Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-10-10T00:37:12.9708633Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-10-10T00:37:12.9708974Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-10-10T00:37:12.9709293Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-10-10T00:37:12.9709625Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-10-10T00:37:12.9709950Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-10-10T00:37:12.9710277Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-10-10T00:37:12.9710600Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-10-10T00:37:12.9710917Z * [new branch] gh/XuehaiPan/384/base -> origin/gh/XuehaiPan/384/base 2025-10-10T00:37:12.9711244Z * [new branch] gh/XuehaiPan/384/head -> origin/gh/XuehaiPan/384/head 2025-10-10T00:37:12.9711570Z * [new branch] gh/XuehaiPan/384/orig -> origin/gh/XuehaiPan/384/orig 2025-10-10T00:37:12.9711894Z * [new branch] gh/XuehaiPan/385/base -> origin/gh/XuehaiPan/385/base 2025-10-10T00:37:12.9712221Z * [new branch] gh/XuehaiPan/385/head -> origin/gh/XuehaiPan/385/head 2025-10-10T00:37:12.9712587Z * [new branch] gh/XuehaiPan/385/orig -> origin/gh/XuehaiPan/385/orig 2025-10-10T00:37:12.9712920Z * [new branch] gh/XuehaiPan/386/base -> origin/gh/XuehaiPan/386/base 2025-10-10T00:37:12.9714685Z * [new branch] gh/XuehaiPan/386/head -> origin/gh/XuehaiPan/386/head 2025-10-10T00:37:12.9715098Z * [new branch] gh/XuehaiPan/386/orig -> origin/gh/XuehaiPan/386/orig 2025-10-10T00:37:12.9720878Z * [new branch] gh/XuehaiPan/387/base -> origin/gh/XuehaiPan/387/base 2025-10-10T00:37:12.9723569Z * [new branch] gh/XuehaiPan/387/head -> origin/gh/XuehaiPan/387/head 2025-10-10T00:37:12.9724075Z * [new branch] gh/XuehaiPan/387/orig -> origin/gh/XuehaiPan/387/orig 2025-10-10T00:37:12.9734300Z * [new branch] gh/XuehaiPan/388/base -> origin/gh/XuehaiPan/388/base 2025-10-10T00:37:12.9738046Z * [new branch] gh/XuehaiPan/388/head -> origin/gh/XuehaiPan/388/head 2025-10-10T00:37:12.9740050Z * [new branch] gh/XuehaiPan/388/orig -> origin/gh/XuehaiPan/388/orig 2025-10-10T00:37:12.9740428Z * [new branch] gh/XuehaiPan/389/base -> origin/gh/XuehaiPan/389/base 2025-10-10T00:37:12.9740769Z * [new branch] gh/XuehaiPan/389/head -> origin/gh/XuehaiPan/389/head 2025-10-10T00:37:12.9741111Z * [new branch] gh/XuehaiPan/389/orig -> origin/gh/XuehaiPan/389/orig 2025-10-10T00:37:12.9741659Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-10-10T00:37:12.9742043Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-10-10T00:37:12.9742407Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-10-10T00:37:12.9742751Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-10-10T00:37:12.9743101Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-10-10T00:37:12.9743442Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-10-10T00:37:12.9743802Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-10-10T00:37:12.9744157Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-10-10T00:37:12.9744522Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-10-10T00:37:12.9744907Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-10-10T00:37:12.9745264Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-10-10T00:37:12.9745631Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-10-10T00:37:12.9745997Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-10-10T00:37:12.9746551Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-10-10T00:37:12.9746925Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-10-10T00:37:12.9747298Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-10-10T00:37:12.9747671Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-10-10T00:37:12.9748058Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-10-10T00:37:12.9748388Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-10-10T00:37:12.9748728Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-10-10T00:37:12.9749067Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-10-10T00:37:12.9749479Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-10-10T00:37:12.9749810Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-10-10T00:37:12.9750136Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-10-10T00:37:12.9750461Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-10-10T00:37:12.9750785Z * [new branch] gh/aakhundov/3/base -> origin/gh/aakhundov/3/base 2025-10-10T00:37:12.9751111Z * [new branch] gh/aakhundov/3/head -> origin/gh/aakhundov/3/head 2025-10-10T00:37:12.9751423Z * [new branch] gh/aakhundov/3/orig -> origin/gh/aakhundov/3/orig 2025-10-10T00:37:12.9751769Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-10-10T00:37:12.9752110Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-10-10T00:37:12.9752441Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-10-10T00:37:12.9752763Z * [new branch] gh/albanD/1/base -> origin/gh/albanD/1/base 2025-10-10T00:37:12.9753067Z * [new branch] gh/albanD/1/head -> origin/gh/albanD/1/head 2025-10-10T00:37:12.9753373Z * [new branch] gh/albanD/1/orig -> origin/gh/albanD/1/orig 2025-10-10T00:37:12.9753671Z * [new branch] gh/albanD/2/base -> origin/gh/albanD/2/base 2025-10-10T00:37:12.9754016Z * [new branch] gh/albanD/2/head -> origin/gh/albanD/2/head 2025-10-10T00:37:12.9754319Z * [new branch] gh/albanD/2/orig -> origin/gh/albanD/2/orig 2025-10-10T00:37:12.9754611Z * [new branch] gh/albanD/3/base -> origin/gh/albanD/3/base 2025-10-10T00:37:12.9754912Z * [new branch] gh/albanD/3/head -> origin/gh/albanD/3/head 2025-10-10T00:37:12.9755218Z * [new branch] gh/albanD/3/orig -> origin/gh/albanD/3/orig 2025-10-10T00:37:12.9755523Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-10-10T00:37:12.9755830Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-10-10T00:37:12.9756140Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-10-10T00:37:12.9756650Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-10-10T00:37:12.9757150Z * [new branch] gh/alexsamardzic/10/base -> origin/gh/alexsamardzic/10/base 2025-10-10T00:37:12.9757516Z * [new branch] gh/alexsamardzic/10/head -> origin/gh/alexsamardzic/10/head 2025-10-10T00:37:12.9757872Z * [new branch] gh/alexsamardzic/10/orig -> origin/gh/alexsamardzic/10/orig 2025-10-10T00:37:12.9763102Z * [new branch] gh/alexsamardzic/11/base -> origin/gh/alexsamardzic/11/base 2025-10-10T00:37:12.9769991Z * [new branch] gh/alexsamardzic/11/head -> origin/gh/alexsamardzic/11/head 2025-10-10T00:37:12.9773025Z * [new branch] gh/alexsamardzic/11/orig -> origin/gh/alexsamardzic/11/orig 2025-10-10T00:37:12.9780026Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-10-10T00:37:12.9781306Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-10-10T00:37:12.9781742Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-10-10T00:37:12.9782134Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-10-10T00:37:12.9782488Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-10-10T00:37:12.9782833Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-10-10T00:37:12.9783574Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-10-10T00:37:12.9783940Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-10-10T00:37:12.9784298Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-10-10T00:37:12.9784652Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-10-10T00:37:12.9785003Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-10-10T00:37:12.9785367Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-10-10T00:37:12.9785729Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-10-10T00:37:12.9786093Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-10-10T00:37:12.9786618Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-10-10T00:37:12.9786986Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-10-10T00:37:12.9787355Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-10-10T00:37:12.9787729Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-10-10T00:37:12.9788088Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-10-10T00:37:12.9788444Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-10-10T00:37:12.9788839Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-10-10T00:37:12.9789202Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-10-10T00:37:12.9789558Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-10-10T00:37:12.9789916Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-10-10T00:37:12.9790276Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-10-10T00:37:12.9790631Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-10-10T00:37:12.9790989Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-10-10T00:37:12.9791340Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-10-10T00:37:12.9791688Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-10-10T00:37:12.9792027Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-10-10T00:37:12.9792374Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-10-10T00:37:12.9792724Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-10-10T00:37:12.9793075Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-10-10T00:37:12.9793431Z * [new branch] gh/angelayi/117/base -> origin/gh/angelayi/117/base 2025-10-10T00:37:12.9793781Z * [new branch] gh/angelayi/117/head -> origin/gh/angelayi/117/head 2025-10-10T00:37:12.9794118Z * [new branch] gh/angelayi/117/orig -> origin/gh/angelayi/117/orig 2025-10-10T00:37:12.9794469Z * [new branch] gh/angelayi/118/base -> origin/gh/angelayi/118/base 2025-10-10T00:37:12.9794820Z * [new branch] gh/angelayi/118/head -> origin/gh/angelayi/118/head 2025-10-10T00:37:12.9795167Z * [new branch] gh/angelayi/118/orig -> origin/gh/angelayi/118/orig 2025-10-10T00:37:12.9795491Z * [new branch] gh/angelayi/119/base -> origin/gh/angelayi/119/base 2025-10-10T00:37:12.9795820Z * [new branch] gh/angelayi/119/head -> origin/gh/angelayi/119/head 2025-10-10T00:37:12.9796210Z * [new branch] gh/angelayi/119/orig -> origin/gh/angelayi/119/orig 2025-10-10T00:37:12.9796544Z * [new branch] gh/angelayi/120/base -> origin/gh/angelayi/120/base 2025-10-10T00:37:12.9796860Z * [new branch] gh/angelayi/120/head -> origin/gh/angelayi/120/head 2025-10-10T00:37:12.9797171Z * [new branch] gh/angelayi/120/orig -> origin/gh/angelayi/120/orig 2025-10-10T00:37:12.9797490Z * [new branch] gh/angelayi/121/base -> origin/gh/angelayi/121/base 2025-10-10T00:37:12.9797812Z * [new branch] gh/angelayi/121/head -> origin/gh/angelayi/121/head 2025-10-10T00:37:12.9798136Z * [new branch] gh/angelayi/121/orig -> origin/gh/angelayi/121/orig 2025-10-10T00:37:12.9799815Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-10-10T00:37:12.9806204Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-10-10T00:37:12.9811012Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-10-10T00:37:12.9815343Z * [new branch] gh/angelayi/123/base -> origin/gh/angelayi/123/base 2025-10-10T00:37:12.9820970Z * [new branch] gh/angelayi/123/head -> origin/gh/angelayi/123/head 2025-10-10T00:37:12.9822859Z * [new branch] gh/angelayi/123/orig -> origin/gh/angelayi/123/orig 2025-10-10T00:37:12.9823313Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-10-10T00:37:12.9830136Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-10-10T00:37:12.9832129Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-10-10T00:37:12.9832604Z * [new branch] gh/angelayi/125/base -> origin/gh/angelayi/125/base 2025-10-10T00:37:12.9832976Z * [new branch] gh/angelayi/125/head -> origin/gh/angelayi/125/head 2025-10-10T00:37:12.9833364Z * [new branch] gh/angelayi/125/orig -> origin/gh/angelayi/125/orig 2025-10-10T00:37:12.9833787Z * [new branch] gh/angelayi/126/base -> origin/gh/angelayi/126/base 2025-10-10T00:37:12.9834142Z * [new branch] gh/angelayi/126/head -> origin/gh/angelayi/126/head 2025-10-10T00:37:12.9834492Z * [new branch] gh/angelayi/126/orig -> origin/gh/angelayi/126/orig 2025-10-10T00:37:12.9834889Z * [new branch] gh/angelayi/127/base -> origin/gh/angelayi/127/base 2025-10-10T00:37:12.9835266Z * [new branch] gh/angelayi/127/head -> origin/gh/angelayi/127/head 2025-10-10T00:37:12.9835619Z * [new branch] gh/angelayi/127/orig -> origin/gh/angelayi/127/orig 2025-10-10T00:37:12.9835978Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-10-10T00:37:12.9836338Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-10-10T00:37:12.9836696Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-10-10T00:37:12.9837056Z * [new branch] gh/angelayi/129/base -> origin/gh/angelayi/129/base 2025-10-10T00:37:12.9837414Z * [new branch] gh/angelayi/129/head -> origin/gh/angelayi/129/head 2025-10-10T00:37:12.9837769Z * [new branch] gh/angelayi/129/orig -> origin/gh/angelayi/129/orig 2025-10-10T00:37:12.9838160Z * [new branch] gh/angelayi/130/base -> origin/gh/angelayi/130/base 2025-10-10T00:37:12.9838518Z * [new branch] gh/angelayi/130/head -> origin/gh/angelayi/130/head 2025-10-10T00:37:12.9838873Z * [new branch] gh/angelayi/130/orig -> origin/gh/angelayi/130/orig 2025-10-10T00:37:12.9839244Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-10-10T00:37:12.9839807Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-10-10T00:37:12.9840181Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-10-10T00:37:12.9840552Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-10-10T00:37:12.9840925Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-10-10T00:37:12.9841296Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-10-10T00:37:12.9841667Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-10-10T00:37:12.9842041Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-10-10T00:37:12.9842406Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-10-10T00:37:12.9842779Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-10-10T00:37:12.9843137Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-10-10T00:37:12.9843470Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-10-10T00:37:12.9843810Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-10-10T00:37:12.9844147Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-10-10T00:37:12.9844546Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-10-10T00:37:12.9844889Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-10-10T00:37:12.9845231Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-10-10T00:37:12.9845566Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-10-10T00:37:12.9845909Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-10-10T00:37:12.9846249Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-10-10T00:37:12.9846591Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-10-10T00:37:12.9846934Z * [new branch] gh/anijain2305/855/base -> origin/gh/anijain2305/855/base 2025-10-10T00:37:12.9847267Z * [new branch] gh/anijain2305/855/head -> origin/gh/anijain2305/855/head 2025-10-10T00:37:12.9847613Z * [new branch] gh/anijain2305/855/orig -> origin/gh/anijain2305/855/orig 2025-10-10T00:37:12.9847954Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-10-10T00:37:12.9848295Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-10-10T00:37:12.9848640Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-10-10T00:37:12.9848978Z * [new branch] gh/anijain2305/867/base -> origin/gh/anijain2305/867/base 2025-10-10T00:37:12.9849320Z * [new branch] gh/anijain2305/867/head -> origin/gh/anijain2305/867/head 2025-10-10T00:37:12.9849656Z * [new branch] gh/anijain2305/867/orig -> origin/gh/anijain2305/867/orig 2025-10-10T00:37:12.9849992Z * [new branch] gh/anijain2305/868/base -> origin/gh/anijain2305/868/base 2025-10-10T00:37:12.9850332Z * [new branch] gh/anijain2305/868/head -> origin/gh/anijain2305/868/head 2025-10-10T00:37:12.9850674Z * [new branch] gh/anijain2305/868/orig -> origin/gh/anijain2305/868/orig 2025-10-10T00:37:12.9851022Z * [new branch] gh/anijain2305/869/base -> origin/gh/anijain2305/869/base 2025-10-10T00:37:12.9851361Z * [new branch] gh/anijain2305/869/head -> origin/gh/anijain2305/869/head 2025-10-10T00:37:12.9851737Z * [new branch] gh/anijain2305/869/orig -> origin/gh/anijain2305/869/orig 2025-10-10T00:37:12.9852076Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-10-10T00:37:12.9852429Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-10-10T00:37:12.9852784Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-10-10T00:37:12.9853153Z * [new branch] gh/anijain2305/871/base -> origin/gh/anijain2305/871/base 2025-10-10T00:37:12.9853560Z * [new branch] gh/anijain2305/871/head -> origin/gh/anijain2305/871/head 2025-10-10T00:37:12.9853899Z * [new branch] gh/anijain2305/871/orig -> origin/gh/anijain2305/871/orig 2025-10-10T00:37:12.9854233Z * [new branch] gh/anijain2305/872/base -> origin/gh/anijain2305/872/base 2025-10-10T00:37:12.9854573Z * [new branch] gh/anijain2305/872/head -> origin/gh/anijain2305/872/head 2025-10-10T00:37:12.9854914Z * [new branch] gh/anijain2305/872/orig -> origin/gh/anijain2305/872/orig 2025-10-10T00:37:12.9855252Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-10-10T00:37:12.9855591Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-10-10T00:37:12.9855923Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-10-10T00:37:12.9856295Z * [new branch] gh/anijain2305/874/base -> origin/gh/anijain2305/874/base 2025-10-10T00:37:12.9856640Z * [new branch] gh/anijain2305/874/head -> origin/gh/anijain2305/874/head 2025-10-10T00:37:12.9856975Z * [new branch] gh/anijain2305/874/orig -> origin/gh/anijain2305/874/orig 2025-10-10T00:37:12.9857330Z * [new branch] gh/anijain2305/875/base -> origin/gh/anijain2305/875/base 2025-10-10T00:37:12.9857680Z * [new branch] gh/anijain2305/875/head -> origin/gh/anijain2305/875/head 2025-10-10T00:37:12.9858034Z * [new branch] gh/anijain2305/875/orig -> origin/gh/anijain2305/875/orig 2025-10-10T00:37:12.9858393Z * [new branch] gh/anijain2305/876/base -> origin/gh/anijain2305/876/base 2025-10-10T00:37:12.9858739Z * [new branch] gh/anijain2305/876/head -> origin/gh/anijain2305/876/head 2025-10-10T00:37:12.9859074Z * [new branch] gh/anijain2305/877/base -> origin/gh/anijain2305/877/base 2025-10-10T00:37:12.9859405Z * [new branch] gh/anijain2305/877/head -> origin/gh/anijain2305/877/head 2025-10-10T00:37:12.9859753Z * [new branch] gh/anijain2305/877/orig -> origin/gh/anijain2305/877/orig 2025-10-10T00:37:12.9860105Z * [new branch] gh/anijain2305/878/base -> origin/gh/anijain2305/878/base 2025-10-10T00:37:12.9860465Z * [new branch] gh/anijain2305/878/head -> origin/gh/anijain2305/878/head 2025-10-10T00:37:12.9860819Z * [new branch] gh/anijain2305/878/orig -> origin/gh/anijain2305/878/orig 2025-10-10T00:37:12.9861165Z * [new branch] gh/anijain2305/879/base -> origin/gh/anijain2305/879/base 2025-10-10T00:37:12.9861686Z * [new branch] gh/anijain2305/879/head -> origin/gh/anijain2305/879/head 2025-10-10T00:37:12.9862287Z * [new branch] gh/anijain2305/879/orig -> origin/gh/anijain2305/879/orig 2025-10-10T00:37:12.9863084Z * [new branch] gh/anijain2305/880/base -> origin/gh/anijain2305/880/base 2025-10-10T00:37:12.9863680Z * [new branch] gh/anijain2305/880/head -> origin/gh/anijain2305/880/head 2025-10-10T00:37:12.9864076Z * [new branch] gh/anijain2305/880/orig -> origin/gh/anijain2305/880/orig 2025-10-10T00:37:12.9864943Z * [new branch] gh/anijain2305/881/base -> origin/gh/anijain2305/881/base 2025-10-10T00:37:12.9865691Z * [new branch] gh/anijain2305/881/head -> origin/gh/anijain2305/881/head 2025-10-10T00:37:12.9867627Z * [new branch] gh/anijain2305/881/orig -> origin/gh/anijain2305/881/orig 2025-10-10T00:37:12.9868164Z * [new branch] gh/anijain2305/882/base -> origin/gh/anijain2305/882/base 2025-10-10T00:37:12.9868667Z * [new branch] gh/anijain2305/882/head -> origin/gh/anijain2305/882/head 2025-10-10T00:37:12.9869172Z * [new branch] gh/anijain2305/882/orig -> origin/gh/anijain2305/882/orig 2025-10-10T00:37:12.9869729Z * [new branch] gh/anijain2305/883/base -> origin/gh/anijain2305/883/base 2025-10-10T00:37:12.9870414Z * [new branch] gh/anijain2305/883/head -> origin/gh/anijain2305/883/head 2025-10-10T00:37:12.9871087Z * [new branch] gh/anijain2305/883/orig -> origin/gh/anijain2305/883/orig 2025-10-10T00:37:12.9872439Z * [new branch] gh/anijain2305/884/base -> origin/gh/anijain2305/884/base 2025-10-10T00:37:12.9876142Z * [new branch] gh/anijain2305/884/head -> origin/gh/anijain2305/884/head 2025-10-10T00:37:12.9876573Z * [new branch] gh/anijain2305/884/orig -> origin/gh/anijain2305/884/orig 2025-10-10T00:37:12.9876947Z * [new branch] gh/anijain2305/885/base -> origin/gh/anijain2305/885/base 2025-10-10T00:37:12.9877311Z * [new branch] gh/anijain2305/885/head -> origin/gh/anijain2305/885/head 2025-10-10T00:37:12.9877802Z * [new branch] gh/anijain2305/885/orig -> origin/gh/anijain2305/885/orig 2025-10-10T00:37:12.9878168Z * [new branch] gh/anijain2305/886/base -> origin/gh/anijain2305/886/base 2025-10-10T00:37:12.9884091Z * [new branch] gh/anijain2305/886/head -> origin/gh/anijain2305/886/head 2025-10-10T00:37:12.9884544Z * [new branch] gh/anijain2305/886/orig -> origin/gh/anijain2305/886/orig 2025-10-10T00:37:12.9884976Z * [new branch] gh/anijain2305/887/base -> origin/gh/anijain2305/887/base 2025-10-10T00:37:12.9885378Z * [new branch] gh/anijain2305/887/head -> origin/gh/anijain2305/887/head 2025-10-10T00:37:12.9885724Z * [new branch] gh/anijain2305/887/orig -> origin/gh/anijain2305/887/orig 2025-10-10T00:37:12.9886131Z * [new branch] gh/anijain2305/888/base -> origin/gh/anijain2305/888/base 2025-10-10T00:37:12.9886537Z * [new branch] gh/anijain2305/888/head -> origin/gh/anijain2305/888/head 2025-10-10T00:37:12.9886956Z * [new branch] gh/anijain2305/888/orig -> origin/gh/anijain2305/888/orig 2025-10-10T00:37:12.9887334Z * [new branch] gh/anijain2305/889/base -> origin/gh/anijain2305/889/base 2025-10-10T00:37:12.9887702Z * [new branch] gh/anijain2305/889/head -> origin/gh/anijain2305/889/head 2025-10-10T00:37:12.9888062Z * [new branch] gh/anijain2305/889/orig -> origin/gh/anijain2305/889/orig 2025-10-10T00:37:12.9888477Z * [new branch] gh/anijain2305/890/base -> origin/gh/anijain2305/890/base 2025-10-10T00:37:12.9888821Z * [new branch] gh/anijain2305/890/head -> origin/gh/anijain2305/890/head 2025-10-10T00:37:12.9889165Z * [new branch] gh/anijain2305/890/orig -> origin/gh/anijain2305/890/orig 2025-10-10T00:37:12.9889512Z * [new branch] gh/anijain2305/891/base -> origin/gh/anijain2305/891/base 2025-10-10T00:37:12.9889853Z * [new branch] gh/anijain2305/891/head -> origin/gh/anijain2305/891/head 2025-10-10T00:37:12.9890206Z * [new branch] gh/anijain2305/891/orig -> origin/gh/anijain2305/891/orig 2025-10-10T00:37:12.9890735Z * [new branch] gh/anijain2305/892/base -> origin/gh/anijain2305/892/base 2025-10-10T00:37:12.9891400Z * [new branch] gh/anijain2305/892/head -> origin/gh/anijain2305/892/head 2025-10-10T00:37:12.9891904Z * [new branch] gh/anijain2305/892/orig -> origin/gh/anijain2305/892/orig 2025-10-10T00:37:12.9892253Z * [new branch] gh/anijain2305/893/base -> origin/gh/anijain2305/893/base 2025-10-10T00:37:12.9897378Z * [new branch] gh/anijain2305/893/head -> origin/gh/anijain2305/893/head 2025-10-10T00:37:12.9897929Z * [new branch] gh/anijain2305/893/orig -> origin/gh/anijain2305/893/orig 2025-10-10T00:37:12.9898438Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-10-10T00:37:12.9898960Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-10-10T00:37:12.9899372Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-10-10T00:37:12.9899747Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-10-10T00:37:12.9900101Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-10-10T00:37:12.9900470Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-10-10T00:37:12.9900832Z * [new branch] gh/anijain2305/896/base -> origin/gh/anijain2305/896/base 2025-10-10T00:37:12.9901198Z * [new branch] gh/anijain2305/896/head -> origin/gh/anijain2305/896/head 2025-10-10T00:37:12.9901557Z * [new branch] gh/anijain2305/896/orig -> origin/gh/anijain2305/896/orig 2025-10-10T00:37:12.9902005Z * [new branch] gh/anijain2305/897/base -> origin/gh/anijain2305/897/base 2025-10-10T00:37:12.9902537Z * [new branch] gh/anijain2305/897/head -> origin/gh/anijain2305/897/head 2025-10-10T00:37:12.9903202Z * [new branch] gh/anijain2305/897/orig -> origin/gh/anijain2305/897/orig 2025-10-10T00:37:12.9904642Z * [new branch] gh/anijain2305/898/base -> origin/gh/anijain2305/898/base 2025-10-10T00:37:12.9905300Z * [new branch] gh/anijain2305/898/head -> origin/gh/anijain2305/898/head 2025-10-10T00:37:12.9906100Z * [new branch] gh/anijain2305/898/orig -> origin/gh/anijain2305/898/orig 2025-10-10T00:37:12.9907764Z * [new branch] gh/anijain2305/899/base -> origin/gh/anijain2305/899/base 2025-10-10T00:37:12.9908542Z * [new branch] gh/anijain2305/899/head -> origin/gh/anijain2305/899/head 2025-10-10T00:37:12.9909379Z * [new branch] gh/anijain2305/899/orig -> origin/gh/anijain2305/899/orig 2025-10-10T00:37:12.9909817Z * [new branch] gh/anijain2305/900/base -> origin/gh/anijain2305/900/base 2025-10-10T00:37:12.9910496Z * [new branch] gh/anijain2305/900/head -> origin/gh/anijain2305/900/head 2025-10-10T00:37:12.9915848Z * [new branch] gh/anijain2305/900/orig -> origin/gh/anijain2305/900/orig 2025-10-10T00:37:12.9921485Z * [new branch] gh/anijain2305/901/base -> origin/gh/anijain2305/901/base 2025-10-10T00:37:12.9926519Z * [new branch] gh/anijain2305/901/head -> origin/gh/anijain2305/901/head 2025-10-10T00:37:12.9927666Z * [new branch] gh/anijain2305/901/orig -> origin/gh/anijain2305/901/orig 2025-10-10T00:37:12.9928052Z * [new branch] gh/anijain2305/902/base -> origin/gh/anijain2305/902/base 2025-10-10T00:37:12.9928442Z * [new branch] gh/anijain2305/902/head -> origin/gh/anijain2305/902/head 2025-10-10T00:37:12.9928818Z * [new branch] gh/anijain2305/902/orig -> origin/gh/anijain2305/902/orig 2025-10-10T00:37:12.9929199Z * [new branch] gh/anijain2305/903/base -> origin/gh/anijain2305/903/base 2025-10-10T00:37:12.9929580Z * [new branch] gh/anijain2305/903/head -> origin/gh/anijain2305/903/head 2025-10-10T00:37:12.9929941Z * [new branch] gh/anijain2305/903/orig -> origin/gh/anijain2305/903/orig 2025-10-10T00:37:12.9930299Z * [new branch] gh/anijain2305/904/base -> origin/gh/anijain2305/904/base 2025-10-10T00:37:12.9930837Z * [new branch] gh/anijain2305/904/head -> origin/gh/anijain2305/904/head 2025-10-10T00:37:12.9931187Z * [new branch] gh/anijain2305/904/orig -> origin/gh/anijain2305/904/orig 2025-10-10T00:37:12.9931762Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-10-10T00:37:12.9932135Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-10-10T00:37:12.9932496Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-10-10T00:37:12.9932866Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-10-10T00:37:12.9933230Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-10-10T00:37:12.9933603Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-10-10T00:37:12.9933985Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-10-10T00:37:12.9934829Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-10-10T00:37:12.9936789Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-10-10T00:37:12.9937377Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-10-10T00:37:12.9937862Z * [new branch] gh/anshul-si/29/base -> origin/gh/anshul-si/29/base 2025-10-10T00:37:12.9938587Z * [new branch] gh/anshul-si/29/head -> origin/gh/anshul-si/29/head 2025-10-10T00:37:12.9939099Z * [new branch] gh/anshul-si/29/orig -> origin/gh/anshul-si/29/orig 2025-10-10T00:37:12.9940061Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-10-10T00:37:12.9940483Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-10-10T00:37:12.9940900Z * [new branch] gh/anshul-si/30/base -> origin/gh/anshul-si/30/base 2025-10-10T00:37:12.9941284Z * [new branch] gh/anshul-si/30/head -> origin/gh/anshul-si/30/head 2025-10-10T00:37:12.9941656Z * [new branch] gh/anshul-si/30/orig -> origin/gh/anshul-si/30/orig 2025-10-10T00:37:12.9942010Z * [new branch] gh/anshul-si/31/base -> origin/gh/anshul-si/31/base 2025-10-10T00:37:12.9942377Z * [new branch] gh/anshul-si/31/head -> origin/gh/anshul-si/31/head 2025-10-10T00:37:12.9942810Z * [new branch] gh/anshul-si/31/orig -> origin/gh/anshul-si/31/orig 2025-10-10T00:37:12.9943167Z * [new branch] gh/anshul-si/32/base -> origin/gh/anshul-si/32/base 2025-10-10T00:37:12.9943527Z * [new branch] gh/anshul-si/32/head -> origin/gh/anshul-si/32/head 2025-10-10T00:37:12.9943904Z * [new branch] gh/anshul-si/32/orig -> origin/gh/anshul-si/32/orig 2025-10-10T00:37:12.9944264Z * [new branch] gh/anshul-si/33/base -> origin/gh/anshul-si/33/base 2025-10-10T00:37:12.9944674Z * [new branch] gh/anshul-si/33/head -> origin/gh/anshul-si/33/head 2025-10-10T00:37:12.9945031Z * [new branch] gh/anshul-si/33/orig -> origin/gh/anshul-si/33/orig 2025-10-10T00:37:12.9945409Z * [new branch] gh/anshul-si/34/base -> origin/gh/anshul-si/34/base 2025-10-10T00:37:12.9945772Z * [new branch] gh/anshul-si/34/head -> origin/gh/anshul-si/34/head 2025-10-10T00:37:12.9946691Z * [new branch] gh/anshul-si/34/orig -> origin/gh/anshul-si/34/orig 2025-10-10T00:37:12.9952104Z * [new branch] gh/anshul-si/35/base -> origin/gh/anshul-si/35/base 2025-10-10T00:37:12.9953958Z * [new branch] gh/anshul-si/35/head -> origin/gh/anshul-si/35/head 2025-10-10T00:37:12.9954903Z * [new branch] gh/anshul-si/35/orig -> origin/gh/anshul-si/35/orig 2025-10-10T00:37:12.9956947Z * [new branch] gh/anshul-si/36/base -> origin/gh/anshul-si/36/base 2025-10-10T00:37:12.9957423Z * [new branch] gh/anshul-si/36/head -> origin/gh/anshul-si/36/head 2025-10-10T00:37:12.9960648Z * [new branch] gh/anshul-si/36/orig -> origin/gh/anshul-si/36/orig 2025-10-10T00:37:12.9961184Z * [new branch] gh/anshul-si/37/base -> origin/gh/anshul-si/37/base 2025-10-10T00:37:12.9961648Z * [new branch] gh/anshul-si/37/head -> origin/gh/anshul-si/37/head 2025-10-10T00:37:12.9961987Z * [new branch] gh/anshul-si/37/orig -> origin/gh/anshul-si/37/orig 2025-10-10T00:37:12.9962329Z * [new branch] gh/anshul-si/38/base -> origin/gh/anshul-si/38/base 2025-10-10T00:37:12.9962651Z * [new branch] gh/anshul-si/38/head -> origin/gh/anshul-si/38/head 2025-10-10T00:37:12.9962979Z * [new branch] gh/anshul-si/38/orig -> origin/gh/anshul-si/38/orig 2025-10-10T00:37:12.9963302Z * [new branch] gh/anshul-si/39/base -> origin/gh/anshul-si/39/base 2025-10-10T00:37:12.9963614Z * [new branch] gh/anshul-si/39/head -> origin/gh/anshul-si/39/head 2025-10-10T00:37:12.9963932Z * [new branch] gh/anshul-si/39/orig -> origin/gh/anshul-si/39/orig 2025-10-10T00:37:12.9964267Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-10-10T00:37:12.9964722Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-10-10T00:37:12.9965045Z * [new branch] gh/anshul-si/40/base -> origin/gh/anshul-si/40/base 2025-10-10T00:37:12.9965367Z * [new branch] gh/anshul-si/40/head -> origin/gh/anshul-si/40/head 2025-10-10T00:37:12.9965686Z * [new branch] gh/anshul-si/40/orig -> origin/gh/anshul-si/40/orig 2025-10-10T00:37:12.9966018Z * [new branch] gh/anshul-si/41/base -> origin/gh/anshul-si/41/base 2025-10-10T00:37:12.9966339Z * [new branch] gh/anshul-si/41/head -> origin/gh/anshul-si/41/head 2025-10-10T00:37:12.9966659Z * [new branch] gh/anshul-si/41/orig -> origin/gh/anshul-si/41/orig 2025-10-10T00:37:12.9966982Z * [new branch] gh/anshul-si/42/base -> origin/gh/anshul-si/42/base 2025-10-10T00:37:12.9967304Z * [new branch] gh/anshul-si/42/head -> origin/gh/anshul-si/42/head 2025-10-10T00:37:12.9967627Z * [new branch] gh/anshul-si/42/orig -> origin/gh/anshul-si/42/orig 2025-10-10T00:37:12.9967948Z * [new branch] gh/anshul-si/43/base -> origin/gh/anshul-si/43/base 2025-10-10T00:37:12.9968267Z * [new branch] gh/anshul-si/43/head -> origin/gh/anshul-si/43/head 2025-10-10T00:37:12.9968748Z * [new branch] gh/anshul-si/43/orig -> origin/gh/anshul-si/43/orig 2025-10-10T00:37:12.9969494Z * [new branch] gh/anshul-si/44/base -> origin/gh/anshul-si/44/base 2025-10-10T00:37:12.9970191Z * [new branch] gh/anshul-si/44/head -> origin/gh/anshul-si/44/head 2025-10-10T00:37:12.9970815Z * [new branch] gh/anshul-si/44/orig -> origin/gh/anshul-si/44/orig 2025-10-10T00:37:12.9973750Z * [new branch] gh/anshul-si/45/base -> origin/gh/anshul-si/45/base 2025-10-10T00:37:12.9974164Z * [new branch] gh/anshul-si/45/head -> origin/gh/anshul-si/45/head 2025-10-10T00:37:12.9974506Z * [new branch] gh/anshul-si/45/orig -> origin/gh/anshul-si/45/orig 2025-10-10T00:37:12.9974830Z * [new branch] gh/anshul-si/46/base -> origin/gh/anshul-si/46/base 2025-10-10T00:37:12.9975327Z * [new branch] gh/anshul-si/46/head -> origin/gh/anshul-si/46/head 2025-10-10T00:37:12.9975819Z * [new branch] gh/anshul-si/46/orig -> origin/gh/anshul-si/46/orig 2025-10-10T00:37:12.9976839Z * [new branch] gh/anshul-si/47/base -> origin/gh/anshul-si/47/base 2025-10-10T00:37:12.9977521Z * [new branch] gh/anshul-si/47/head -> origin/gh/anshul-si/47/head 2025-10-10T00:37:12.9978178Z * [new branch] gh/anshul-si/47/orig -> origin/gh/anshul-si/47/orig 2025-10-10T00:37:12.9979258Z * [new branch] gh/anshul-si/48/base -> origin/gh/anshul-si/48/base 2025-10-10T00:37:12.9979598Z * [new branch] gh/anshul-si/48/head -> origin/gh/anshul-si/48/head 2025-10-10T00:37:12.9980390Z * [new branch] gh/anshul-si/48/orig -> origin/gh/anshul-si/48/orig 2025-10-10T00:37:12.9981844Z * [new branch] gh/anshul-si/49/base -> origin/gh/anshul-si/49/base 2025-10-10T00:37:12.9982188Z * [new branch] gh/anshul-si/49/head -> origin/gh/anshul-si/49/head 2025-10-10T00:37:12.9982766Z * [new branch] gh/anshul-si/49/orig -> origin/gh/anshul-si/49/orig 2025-10-10T00:37:12.9983667Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-10-10T00:37:12.9984246Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-10-10T00:37:12.9989144Z * [new branch] gh/anshul-si/50/base -> origin/gh/anshul-si/50/base 2025-10-10T00:37:12.9989533Z * [new branch] gh/anshul-si/50/head -> origin/gh/anshul-si/50/head 2025-10-10T00:37:12.9990028Z * [new branch] gh/anshul-si/50/orig -> origin/gh/anshul-si/50/orig 2025-10-10T00:37:12.9990351Z * [new branch] gh/anshul-si/51/base -> origin/gh/anshul-si/51/base 2025-10-10T00:37:12.9990676Z * [new branch] gh/anshul-si/51/head -> origin/gh/anshul-si/51/head 2025-10-10T00:37:12.9990997Z * [new branch] gh/anshul-si/51/orig -> origin/gh/anshul-si/51/orig 2025-10-10T00:37:12.9991322Z * [new branch] gh/anshul-si/52/base -> origin/gh/anshul-si/52/base 2025-10-10T00:37:12.9991642Z * [new branch] gh/anshul-si/52/head -> origin/gh/anshul-si/52/head 2025-10-10T00:37:12.9996865Z * [new branch] gh/anshul-si/52/orig -> origin/gh/anshul-si/52/orig 2025-10-10T00:37:12.9997262Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-10-10T00:37:12.9997603Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-10-10T00:37:12.9997948Z * [new branch] gh/aorenste/133/base -> origin/gh/aorenste/133/base 2025-10-10T00:37:12.9998271Z * [new branch] gh/aorenste/133/head -> origin/gh/aorenste/133/head 2025-10-10T00:37:12.9998588Z * [new branch] gh/aorenste/133/orig -> origin/gh/aorenste/133/orig 2025-10-10T00:37:12.9998899Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-10-10T00:37:12.9999225Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-10-10T00:37:12.9999556Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-10-10T00:37:12.9999893Z * [new branch] gh/aorenste/135/base -> origin/gh/aorenste/135/base 2025-10-10T00:37:13.0000211Z * [new branch] gh/aorenste/135/head -> origin/gh/aorenste/135/head 2025-10-10T00:37:13.0000522Z * [new branch] gh/aorenste/135/orig -> origin/gh/aorenste/135/orig 2025-10-10T00:37:13.0000848Z * [new branch] gh/aorenste/136/base -> origin/gh/aorenste/136/base 2025-10-10T00:37:13.0001552Z * [new branch] gh/aorenste/136/head -> origin/gh/aorenste/136/head 2025-10-10T00:37:13.0002116Z * [new branch] gh/aorenste/136/orig -> origin/gh/aorenste/136/orig 2025-10-10T00:37:13.0003310Z * [new branch] gh/aorenste/137/base -> origin/gh/aorenste/137/base 2025-10-10T00:37:13.0008634Z * [new branch] gh/aorenste/137/head -> origin/gh/aorenste/137/head 2025-10-10T00:37:13.0009038Z * [new branch] gh/aorenste/137/orig -> origin/gh/aorenste/137/orig 2025-10-10T00:37:13.0009412Z * [new branch] gh/aorenste/138/base -> origin/gh/aorenste/138/base 2025-10-10T00:37:13.0009755Z * [new branch] gh/aorenste/138/head -> origin/gh/aorenste/138/head 2025-10-10T00:37:13.0010104Z * [new branch] gh/aorenste/138/orig -> origin/gh/aorenste/138/orig 2025-10-10T00:37:13.0010436Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-10-10T00:37:13.0010768Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-10-10T00:37:13.0011106Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-10-10T00:37:13.0011475Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-10-10T00:37:13.0011847Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-10-10T00:37:13.0012212Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-10-10T00:37:13.0012569Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-10-10T00:37:13.0013283Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-10-10T00:37:13.0017887Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-10-10T00:37:13.0018302Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-10-10T00:37:13.0018643Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-10-10T00:37:13.0018968Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-10-10T00:37:13.0019304Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-10-10T00:37:13.0019631Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-10-10T00:37:13.0019962Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-10-10T00:37:13.0020325Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-10-10T00:37:13.0021150Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-10-10T00:37:13.0022188Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-10-10T00:37:13.0022646Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-10-10T00:37:13.0023393Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-10-10T00:37:13.0024756Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-10-10T00:37:13.0025122Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-10-10T00:37:13.0025750Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-10-10T00:37:13.0031376Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-10-10T00:37:13.0031938Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-10-10T00:37:13.0032280Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-10-10T00:37:13.0032628Z * [new branch] gh/bdhirsh/671/base -> origin/gh/bdhirsh/671/base 2025-10-10T00:37:13.0032958Z * [new branch] gh/bdhirsh/671/head -> origin/gh/bdhirsh/671/head 2025-10-10T00:37:13.0033281Z * [new branch] gh/bdhirsh/671/orig -> origin/gh/bdhirsh/671/orig 2025-10-10T00:37:13.0033608Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-10-10T00:37:13.0038965Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-10-10T00:37:13.0041610Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-10-10T00:37:13.0042309Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-10-10T00:37:13.0042762Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-10-10T00:37:13.0043187Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-10-10T00:37:13.0043653Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-10-10T00:37:13.0044053Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-10-10T00:37:13.0044405Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-10-10T00:37:13.0044821Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-10-10T00:37:13.0045225Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-10-10T00:37:13.0045628Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-10-10T00:37:13.0046029Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-10-10T00:37:13.0051778Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-10-10T00:37:13.0053941Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-10-10T00:37:13.0054538Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-10-10T00:37:13.0059237Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-10-10T00:37:13.0060874Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-10-10T00:37:13.0061271Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-10-10T00:37:13.0061670Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-10-10T00:37:13.0062056Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-10-10T00:37:13.0062453Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-10-10T00:37:13.0062881Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-10-10T00:37:13.0063286Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-10-10T00:37:13.0063704Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-10-10T00:37:13.0064108Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-10-10T00:37:13.0064531Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-10-10T00:37:13.0064945Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-10-10T00:37:13.0065357Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-10-10T00:37:13.0065755Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-10-10T00:37:13.0066146Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-10-10T00:37:13.0066861Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-10-10T00:37:13.0067245Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-10-10T00:37:13.0067624Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-10-10T00:37:13.0068158Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-10-10T00:37:13.0068508Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-10-10T00:37:13.0068846Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-10-10T00:37:13.0069177Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-10-10T00:37:13.0069513Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-10-10T00:37:13.0069867Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-10-10T00:37:13.0070229Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-10-10T00:37:13.0070591Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-10-10T00:37:13.0070969Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-10-10T00:37:13.0071332Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-10-10T00:37:13.0071699Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-10-10T00:37:13.0072050Z * [new branch] gh/bobrenjc93/553/base -> origin/gh/bobrenjc93/553/base 2025-10-10T00:37:13.0072396Z * [new branch] gh/bobrenjc93/553/head -> origin/gh/bobrenjc93/553/head 2025-10-10T00:37:13.0072720Z * [new branch] gh/bobrenjc93/553/orig -> origin/gh/bobrenjc93/553/orig 2025-10-10T00:37:13.0073089Z * [new branch] gh/bobrenjc93/554/base -> origin/gh/bobrenjc93/554/base 2025-10-10T00:37:13.0073418Z * [new branch] gh/bobrenjc93/554/head -> origin/gh/bobrenjc93/554/head 2025-10-10T00:37:13.0073755Z * [new branch] gh/bobrenjc93/554/orig -> origin/gh/bobrenjc93/554/orig 2025-10-10T00:37:13.0074263Z * [new branch] gh/bobrenjc93/555/base -> origin/gh/bobrenjc93/555/base 2025-10-10T00:37:13.0074717Z * [new branch] gh/bobrenjc93/555/head -> origin/gh/bobrenjc93/555/head 2025-10-10T00:37:13.0075136Z * [new branch] gh/bobrenjc93/555/orig -> origin/gh/bobrenjc93/555/orig 2025-10-10T00:37:13.0081040Z * [new branch] gh/bobrenjc93/557/base -> origin/gh/bobrenjc93/557/base 2025-10-10T00:37:13.0083539Z * [new branch] gh/bobrenjc93/557/head -> origin/gh/bobrenjc93/557/head 2025-10-10T00:37:13.0084170Z * [new branch] gh/bobrenjc93/557/orig -> origin/gh/bobrenjc93/557/orig 2025-10-10T00:37:13.0084681Z * [new branch] gh/bobrenjc93/558/base -> origin/gh/bobrenjc93/558/base 2025-10-10T00:37:13.0085179Z * [new branch] gh/bobrenjc93/558/head -> origin/gh/bobrenjc93/558/head 2025-10-10T00:37:13.0085662Z * [new branch] gh/bobrenjc93/558/orig -> origin/gh/bobrenjc93/558/orig 2025-10-10T00:37:13.0086159Z * [new branch] gh/bobrenjc93/559/base -> origin/gh/bobrenjc93/559/base 2025-10-10T00:37:13.0086527Z * [new branch] gh/bobrenjc93/559/head -> origin/gh/bobrenjc93/559/head 2025-10-10T00:37:13.0086906Z * [new branch] gh/bobrenjc93/559/orig -> origin/gh/bobrenjc93/559/orig 2025-10-10T00:37:13.0087266Z * [new branch] gh/bobrenjc93/560/base -> origin/gh/bobrenjc93/560/base 2025-10-10T00:37:13.0087622Z * [new branch] gh/bobrenjc93/560/head -> origin/gh/bobrenjc93/560/head 2025-10-10T00:37:13.0087982Z * [new branch] gh/bobrenjc93/560/orig -> origin/gh/bobrenjc93/560/orig 2025-10-10T00:37:13.0088340Z * [new branch] gh/bobrenjc93/561/base -> origin/gh/bobrenjc93/561/base 2025-10-10T00:37:13.0088696Z * [new branch] gh/bobrenjc93/561/head -> origin/gh/bobrenjc93/561/head 2025-10-10T00:37:13.0089051Z * [new branch] gh/bobrenjc93/561/orig -> origin/gh/bobrenjc93/561/orig 2025-10-10T00:37:13.0089587Z * [new branch] gh/bobrenjc93/562/base -> origin/gh/bobrenjc93/562/base 2025-10-10T00:37:13.0089943Z * [new branch] gh/bobrenjc93/562/head -> origin/gh/bobrenjc93/562/head 2025-10-10T00:37:13.0090306Z * [new branch] gh/bobrenjc93/562/orig -> origin/gh/bobrenjc93/562/orig 2025-10-10T00:37:13.0090681Z * [new branch] gh/bobrenjc93/563/base -> origin/gh/bobrenjc93/563/base 2025-10-10T00:37:13.0091040Z * [new branch] gh/bobrenjc93/563/head -> origin/gh/bobrenjc93/563/head 2025-10-10T00:37:13.0091401Z * [new branch] gh/bobrenjc93/563/orig -> origin/gh/bobrenjc93/563/orig 2025-10-10T00:37:13.0091752Z * [new branch] gh/bobrenjc93/564/base -> origin/gh/bobrenjc93/564/base 2025-10-10T00:37:13.0092117Z * [new branch] gh/bobrenjc93/564/head -> origin/gh/bobrenjc93/564/head 2025-10-10T00:37:13.0092481Z * [new branch] gh/bobrenjc93/564/orig -> origin/gh/bobrenjc93/564/orig 2025-10-10T00:37:13.0094170Z * [new branch] gh/bobrenjc93/565/base -> origin/gh/bobrenjc93/565/base 2025-10-10T00:37:13.0094676Z * [new branch] gh/bobrenjc93/565/head -> origin/gh/bobrenjc93/565/head 2025-10-10T00:37:13.0095158Z * [new branch] gh/bobrenjc93/565/orig -> origin/gh/bobrenjc93/565/orig 2025-10-10T00:37:13.0095331Z * [new branch] gh/bobrenjc93/566/base -> origin/gh/bobrenjc93/566/base 2025-10-10T00:37:13.0095855Z * [new branch] gh/bobrenjc93/566/head -> origin/gh/bobrenjc93/566/head 2025-10-10T00:37:13.0096812Z * [new branch] gh/bobrenjc93/566/orig -> origin/gh/bobrenjc93/566/orig 2025-10-10T00:37:13.0097578Z * [new branch] gh/bobrenjc93/567/base -> origin/gh/bobrenjc93/567/base 2025-10-10T00:37:13.0098127Z * [new branch] gh/bobrenjc93/567/head -> origin/gh/bobrenjc93/567/head 2025-10-10T00:37:13.0099167Z * [new branch] gh/bobrenjc93/567/orig -> origin/gh/bobrenjc93/567/orig 2025-10-10T00:37:13.0099739Z * [new branch] gh/bobrenjc93/568/base -> origin/gh/bobrenjc93/568/base 2025-10-10T00:37:13.0100666Z * [new branch] gh/bobrenjc93/568/head -> origin/gh/bobrenjc93/568/head 2025-10-10T00:37:13.0101097Z * [new branch] gh/bobrenjc93/568/orig -> origin/gh/bobrenjc93/568/orig 2025-10-10T00:37:13.0102228Z * [new branch] gh/bobrenjc93/569/base -> origin/gh/bobrenjc93/569/base 2025-10-10T00:37:13.0102880Z * [new branch] gh/bobrenjc93/569/head -> origin/gh/bobrenjc93/569/head 2025-10-10T00:37:13.0103602Z * [new branch] gh/bobrenjc93/569/orig -> origin/gh/bobrenjc93/569/orig 2025-10-10T00:37:13.0104693Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-10-10T00:37:13.0105268Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-10-10T00:37:13.0105707Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-10-10T00:37:13.0107077Z * [new branch] gh/bobrenjc93/571/base -> origin/gh/bobrenjc93/571/base 2025-10-10T00:37:13.0107687Z * [new branch] gh/bobrenjc93/571/head -> origin/gh/bobrenjc93/571/head 2025-10-10T00:37:13.0108382Z * [new branch] gh/bobrenjc93/571/orig -> origin/gh/bobrenjc93/571/orig 2025-10-10T00:37:13.0109513Z * [new branch] gh/bobrenjc93/572/base -> origin/gh/bobrenjc93/572/base 2025-10-10T00:37:13.0110003Z * [new branch] gh/bobrenjc93/572/head -> origin/gh/bobrenjc93/572/head 2025-10-10T00:37:13.0111025Z * [new branch] gh/bobrenjc93/572/orig -> origin/gh/bobrenjc93/572/orig 2025-10-10T00:37:13.0111588Z * [new branch] gh/bobrenjc93/573/base -> origin/gh/bobrenjc93/573/base 2025-10-10T00:37:13.0112982Z * [new branch] gh/bobrenjc93/573/head -> origin/gh/bobrenjc93/573/head 2025-10-10T00:37:13.0113310Z * [new branch] gh/bobrenjc93/573/orig -> origin/gh/bobrenjc93/573/orig 2025-10-10T00:37:13.0114592Z * [new branch] gh/bobrenjc93/574/base -> origin/gh/bobrenjc93/574/base 2025-10-10T00:37:13.0114902Z * [new branch] gh/bobrenjc93/574/head -> origin/gh/bobrenjc93/574/head 2025-10-10T00:37:13.0115944Z * [new branch] gh/bobrenjc93/574/orig -> origin/gh/bobrenjc93/574/orig 2025-10-10T00:37:13.0120421Z * [new branch] gh/bobrenjc93/575/base -> origin/gh/bobrenjc93/575/base 2025-10-10T00:37:13.0120603Z * [new branch] gh/bobrenjc93/575/head -> origin/gh/bobrenjc93/575/head 2025-10-10T00:37:13.0120753Z * [new branch] gh/bobrenjc93/575/orig -> origin/gh/bobrenjc93/575/orig 2025-10-10T00:37:13.0120910Z * [new branch] gh/bobrenjc93/576/base -> origin/gh/bobrenjc93/576/base 2025-10-10T00:37:13.0121083Z * [new branch] gh/bobrenjc93/576/head -> origin/gh/bobrenjc93/576/head 2025-10-10T00:37:13.0121246Z * [new branch] gh/bobrenjc93/576/orig -> origin/gh/bobrenjc93/576/orig 2025-10-10T00:37:13.0121435Z * [new branch] gh/bobrenjc93/577/base -> origin/gh/bobrenjc93/577/base 2025-10-10T00:37:13.0122369Z * [new branch] gh/bobrenjc93/577/head -> origin/gh/bobrenjc93/577/head 2025-10-10T00:37:13.0127571Z * [new branch] gh/bobrenjc93/577/orig -> origin/gh/bobrenjc93/577/orig 2025-10-10T00:37:13.0127775Z * [new branch] gh/bobrenjc93/578/base -> origin/gh/bobrenjc93/578/base 2025-10-10T00:37:13.0127925Z * [new branch] gh/bobrenjc93/578/head -> origin/gh/bobrenjc93/578/head 2025-10-10T00:37:13.0128070Z * [new branch] gh/bobrenjc93/578/orig -> origin/gh/bobrenjc93/578/orig 2025-10-10T00:37:13.0128241Z * [new branch] gh/bobrenjc93/579/base -> origin/gh/bobrenjc93/579/base 2025-10-10T00:37:13.0128397Z * [new branch] gh/bobrenjc93/579/head -> origin/gh/bobrenjc93/579/head 2025-10-10T00:37:13.0128553Z * [new branch] gh/bobrenjc93/579/orig -> origin/gh/bobrenjc93/579/orig 2025-10-10T00:37:13.0133898Z * [new branch] gh/bobrenjc93/580/base -> origin/gh/bobrenjc93/580/base 2025-10-10T00:37:13.0139536Z * [new branch] gh/bobrenjc93/580/head -> origin/gh/bobrenjc93/580/head 2025-10-10T00:37:13.0139754Z * [new branch] gh/bobrenjc93/580/orig -> origin/gh/bobrenjc93/580/orig 2025-10-10T00:37:13.0139922Z * [new branch] gh/bobrenjc93/581/base -> origin/gh/bobrenjc93/581/base 2025-10-10T00:37:13.0140078Z * [new branch] gh/bobrenjc93/581/head -> origin/gh/bobrenjc93/581/head 2025-10-10T00:37:13.0140232Z * [new branch] gh/bobrenjc93/581/orig -> origin/gh/bobrenjc93/581/orig 2025-10-10T00:37:13.0140405Z * [new branch] gh/bobrenjc93/582/base -> origin/gh/bobrenjc93/582/base 2025-10-10T00:37:13.0140559Z * [new branch] gh/bobrenjc93/582/head -> origin/gh/bobrenjc93/582/head 2025-10-10T00:37:13.0140717Z * [new branch] gh/bobrenjc93/582/orig -> origin/gh/bobrenjc93/582/orig 2025-10-10T00:37:13.0140866Z * [new branch] gh/bobrenjc93/583/base -> origin/gh/bobrenjc93/583/base 2025-10-10T00:37:13.0141017Z * [new branch] gh/bobrenjc93/583/head -> origin/gh/bobrenjc93/583/head 2025-10-10T00:37:13.0141185Z * [new branch] gh/bobrenjc93/583/orig -> origin/gh/bobrenjc93/583/orig 2025-10-10T00:37:13.0141339Z * [new branch] gh/bobrenjc93/584/base -> origin/gh/bobrenjc93/584/base 2025-10-10T00:37:13.0141500Z * [new branch] gh/bobrenjc93/584/head -> origin/gh/bobrenjc93/584/head 2025-10-10T00:37:13.0141851Z * [new branch] gh/bobrenjc93/584/orig -> origin/gh/bobrenjc93/584/orig 2025-10-10T00:37:13.0142017Z * [new branch] gh/bobrenjc93/585/base -> origin/gh/bobrenjc93/585/base 2025-10-10T00:37:13.0142169Z * [new branch] gh/bobrenjc93/585/head -> origin/gh/bobrenjc93/585/head 2025-10-10T00:37:13.0142321Z * [new branch] gh/bobrenjc93/585/orig -> origin/gh/bobrenjc93/585/orig 2025-10-10T00:37:13.0142946Z * [new branch] gh/bobrenjc93/586/base -> origin/gh/bobrenjc93/586/base 2025-10-10T00:37:13.0143497Z * [new branch] gh/bobrenjc93/586/head -> origin/gh/bobrenjc93/586/head 2025-10-10T00:37:13.0143946Z * [new branch] gh/bobrenjc93/586/orig -> origin/gh/bobrenjc93/586/orig 2025-10-10T00:37:13.0145509Z * [new branch] gh/bobrenjc93/587/base -> origin/gh/bobrenjc93/587/base 2025-10-10T00:37:13.0145681Z * [new branch] gh/bobrenjc93/587/head -> origin/gh/bobrenjc93/587/head 2025-10-10T00:37:13.0146539Z * [new branch] gh/bobrenjc93/587/orig -> origin/gh/bobrenjc93/587/orig 2025-10-10T00:37:13.0151382Z * [new branch] gh/bobrenjc93/588/base -> origin/gh/bobrenjc93/588/base 2025-10-10T00:37:13.0151615Z * [new branch] gh/bobrenjc93/588/head -> origin/gh/bobrenjc93/588/head 2025-10-10T00:37:13.0151810Z * [new branch] gh/bobrenjc93/588/orig -> origin/gh/bobrenjc93/588/orig 2025-10-10T00:37:13.0151989Z * [new branch] gh/bobrenjc93/589/base -> origin/gh/bobrenjc93/589/base 2025-10-10T00:37:13.0152430Z * [new branch] gh/bobrenjc93/589/head -> origin/gh/bobrenjc93/589/head 2025-10-10T00:37:13.0152665Z * [new branch] gh/bobrenjc93/589/orig -> origin/gh/bobrenjc93/589/orig 2025-10-10T00:37:13.0160318Z * [new branch] gh/bobrenjc93/590/base -> origin/gh/bobrenjc93/590/base 2025-10-10T00:37:13.0165059Z * [new branch] gh/bobrenjc93/590/head -> origin/gh/bobrenjc93/590/head 2025-10-10T00:37:13.0170250Z * [new branch] gh/bobrenjc93/590/orig -> origin/gh/bobrenjc93/590/orig 2025-10-10T00:37:13.0170441Z * [new branch] gh/bobrenjc93/591/base -> origin/gh/bobrenjc93/591/base 2025-10-10T00:37:13.0170633Z * [new branch] gh/bobrenjc93/591/head -> origin/gh/bobrenjc93/591/head 2025-10-10T00:37:13.0170800Z * [new branch] gh/bobrenjc93/591/orig -> origin/gh/bobrenjc93/591/orig 2025-10-10T00:37:13.0170993Z * [new branch] gh/bobrenjc93/592/base -> origin/gh/bobrenjc93/592/base 2025-10-10T00:37:13.0171155Z * [new branch] gh/bobrenjc93/592/head -> origin/gh/bobrenjc93/592/head 2025-10-10T00:37:13.0171331Z * [new branch] gh/bobrenjc93/592/orig -> origin/gh/bobrenjc93/592/orig 2025-10-10T00:37:13.0171501Z * [new branch] gh/bobrenjc93/593/base -> origin/gh/bobrenjc93/593/base 2025-10-10T00:37:13.0171665Z * [new branch] gh/bobrenjc93/593/head -> origin/gh/bobrenjc93/593/head 2025-10-10T00:37:13.0171829Z * [new branch] gh/bobrenjc93/593/orig -> origin/gh/bobrenjc93/593/orig 2025-10-10T00:37:13.0171971Z * [new branch] gh/bobrenjc93/594/base -> origin/gh/bobrenjc93/594/base 2025-10-10T00:37:13.0172137Z * [new branch] gh/bobrenjc93/594/head -> origin/gh/bobrenjc93/594/head 2025-10-10T00:37:13.0172299Z * [new branch] gh/bobrenjc93/594/orig -> origin/gh/bobrenjc93/594/orig 2025-10-10T00:37:13.0172481Z * [new branch] gh/bobrenjc93/595/base -> origin/gh/bobrenjc93/595/base 2025-10-10T00:37:13.0172635Z * [new branch] gh/bobrenjc93/595/head -> origin/gh/bobrenjc93/595/head 2025-10-10T00:37:13.0172790Z * [new branch] gh/bobrenjc93/595/orig -> origin/gh/bobrenjc93/595/orig 2025-10-10T00:37:13.0172954Z * [new branch] gh/bobrenjc93/596/base -> origin/gh/bobrenjc93/596/base 2025-10-10T00:37:13.0173269Z * [new branch] gh/bobrenjc93/596/head -> origin/gh/bobrenjc93/596/head 2025-10-10T00:37:13.0173443Z * [new branch] gh/bobrenjc93/596/orig -> origin/gh/bobrenjc93/596/orig 2025-10-10T00:37:13.0173605Z * [new branch] gh/bobrenjc93/597/base -> origin/gh/bobrenjc93/597/base 2025-10-10T00:37:13.0173791Z * [new branch] gh/bobrenjc93/597/head -> origin/gh/bobrenjc93/597/head 2025-10-10T00:37:13.0173967Z * [new branch] gh/bobrenjc93/597/orig -> origin/gh/bobrenjc93/597/orig 2025-10-10T00:37:13.0174138Z * [new branch] gh/bobrenjc93/598/base -> origin/gh/bobrenjc93/598/base 2025-10-10T00:37:13.0174442Z * [new branch] gh/bobrenjc93/598/head -> origin/gh/bobrenjc93/598/head 2025-10-10T00:37:13.0174617Z * [new branch] gh/bobrenjc93/598/orig -> origin/gh/bobrenjc93/598/orig 2025-10-10T00:37:13.0174787Z * [new branch] gh/bobrenjc93/599/base -> origin/gh/bobrenjc93/599/base 2025-10-10T00:37:13.0174942Z * [new branch] gh/bobrenjc93/599/head -> origin/gh/bobrenjc93/599/head 2025-10-10T00:37:13.0175081Z * [new branch] gh/bobrenjc93/599/orig -> origin/gh/bobrenjc93/599/orig 2025-10-10T00:37:13.0181294Z * [new branch] gh/bobrenjc93/600/base -> origin/gh/bobrenjc93/600/base 2025-10-10T00:37:13.0184769Z * [new branch] gh/bobrenjc93/600/head -> origin/gh/bobrenjc93/600/head 2025-10-10T00:37:13.0185493Z * [new branch] gh/bobrenjc93/600/orig -> origin/gh/bobrenjc93/600/orig 2025-10-10T00:37:13.0185696Z * [new branch] gh/bobrenjc93/601/base -> origin/gh/bobrenjc93/601/base 2025-10-10T00:37:13.0185864Z * [new branch] gh/bobrenjc93/601/head -> origin/gh/bobrenjc93/601/head 2025-10-10T00:37:13.0186026Z * [new branch] gh/bobrenjc93/601/orig -> origin/gh/bobrenjc93/601/orig 2025-10-10T00:37:13.0186206Z * [new branch] gh/bobrenjc93/602/base -> origin/gh/bobrenjc93/602/base 2025-10-10T00:37:13.0186593Z * [new branch] gh/bobrenjc93/602/head -> origin/gh/bobrenjc93/602/head 2025-10-10T00:37:13.0186753Z * [new branch] gh/bobrenjc93/602/orig -> origin/gh/bobrenjc93/602/orig 2025-10-10T00:37:13.0186906Z * [new branch] gh/bobrenjc93/603/base -> origin/gh/bobrenjc93/603/base 2025-10-10T00:37:13.0187060Z * [new branch] gh/bobrenjc93/603/head -> origin/gh/bobrenjc93/603/head 2025-10-10T00:37:13.0187224Z * [new branch] gh/bobrenjc93/603/orig -> origin/gh/bobrenjc93/603/orig 2025-10-10T00:37:13.0191220Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-10-10T00:37:13.0193882Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-10-10T00:37:13.0194163Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-10-10T00:37:13.0199371Z * [new branch] gh/bobrenjc93/605/base -> origin/gh/bobrenjc93/605/base 2025-10-10T00:37:13.0204461Z * [new branch] gh/bobrenjc93/605/head -> origin/gh/bobrenjc93/605/head 2025-10-10T00:37:13.0209558Z * [new branch] gh/bobrenjc93/605/orig -> origin/gh/bobrenjc93/605/orig 2025-10-10T00:37:13.0211614Z * [new branch] gh/bobrenjc93/606/base -> origin/gh/bobrenjc93/606/base 2025-10-10T00:37:13.0217474Z * [new branch] gh/bobrenjc93/606/head -> origin/gh/bobrenjc93/606/head 2025-10-10T00:37:13.0219863Z * [new branch] gh/bobrenjc93/606/orig -> origin/gh/bobrenjc93/606/orig 2025-10-10T00:37:13.0220178Z * [new branch] gh/bobrenjc93/607/base -> origin/gh/bobrenjc93/607/base 2025-10-10T00:37:13.0220439Z * [new branch] gh/bobrenjc93/607/head -> origin/gh/bobrenjc93/607/head 2025-10-10T00:37:13.0220932Z * [new branch] gh/bobrenjc93/607/orig -> origin/gh/bobrenjc93/607/orig 2025-10-10T00:37:13.0221276Z * [new branch] gh/bobrenjc93/608/base -> origin/gh/bobrenjc93/608/base 2025-10-10T00:37:13.0221539Z * [new branch] gh/bobrenjc93/608/head -> origin/gh/bobrenjc93/608/head 2025-10-10T00:37:13.0221735Z * [new branch] gh/bobrenjc93/608/orig -> origin/gh/bobrenjc93/608/orig 2025-10-10T00:37:13.0221906Z * [new branch] gh/bobrenjc93/609/base -> origin/gh/bobrenjc93/609/base 2025-10-10T00:37:13.0222069Z * [new branch] gh/bobrenjc93/609/head -> origin/gh/bobrenjc93/609/head 2025-10-10T00:37:13.0222230Z * [new branch] gh/bobrenjc93/609/orig -> origin/gh/bobrenjc93/609/orig 2025-10-10T00:37:13.0222390Z * [new branch] gh/bobrenjc93/610/base -> origin/gh/bobrenjc93/610/base 2025-10-10T00:37:13.0222556Z * [new branch] gh/bobrenjc93/610/head -> origin/gh/bobrenjc93/610/head 2025-10-10T00:37:13.0222720Z * [new branch] gh/bobrenjc93/610/orig -> origin/gh/bobrenjc93/610/orig 2025-10-10T00:37:13.0222881Z * [new branch] gh/bobrenjc93/611/base -> origin/gh/bobrenjc93/611/base 2025-10-10T00:37:13.0223050Z * [new branch] gh/bobrenjc93/611/head -> origin/gh/bobrenjc93/611/head 2025-10-10T00:37:13.0223206Z * [new branch] gh/bobrenjc93/611/orig -> origin/gh/bobrenjc93/611/orig 2025-10-10T00:37:13.0223369Z * [new branch] gh/bobrenjc93/612/base -> origin/gh/bobrenjc93/612/base 2025-10-10T00:37:13.0223579Z * [new branch] gh/bobrenjc93/612/head -> origin/gh/bobrenjc93/612/head 2025-10-10T00:37:13.0223750Z * [new branch] gh/bobrenjc93/612/orig -> origin/gh/bobrenjc93/612/orig 2025-10-10T00:37:13.0223911Z * [new branch] gh/bobrenjc93/613/base -> origin/gh/bobrenjc93/613/base 2025-10-10T00:37:13.0224069Z * [new branch] gh/bobrenjc93/613/head -> origin/gh/bobrenjc93/613/head 2025-10-10T00:37:13.0224235Z * [new branch] gh/bobrenjc93/613/orig -> origin/gh/bobrenjc93/613/orig 2025-10-10T00:37:13.0224392Z * [new branch] gh/bobrenjc93/614/base -> origin/gh/bobrenjc93/614/base 2025-10-10T00:37:13.0224554Z * [new branch] gh/bobrenjc93/614/head -> origin/gh/bobrenjc93/614/head 2025-10-10T00:37:13.0224711Z * [new branch] gh/bobrenjc93/614/orig -> origin/gh/bobrenjc93/614/orig 2025-10-10T00:37:13.0224872Z * [new branch] gh/bobrenjc93/615/base -> origin/gh/bobrenjc93/615/base 2025-10-10T00:37:13.0225031Z * [new branch] gh/bobrenjc93/615/head -> origin/gh/bobrenjc93/615/head 2025-10-10T00:37:13.0225187Z * [new branch] gh/bobrenjc93/615/orig -> origin/gh/bobrenjc93/615/orig 2025-10-10T00:37:13.0225348Z * [new branch] gh/bobrenjc93/616/base -> origin/gh/bobrenjc93/616/base 2025-10-10T00:37:13.0225514Z * [new branch] gh/bobrenjc93/616/head -> origin/gh/bobrenjc93/616/head 2025-10-10T00:37:13.0225680Z * [new branch] gh/bobrenjc93/616/orig -> origin/gh/bobrenjc93/616/orig 2025-10-10T00:37:13.0225841Z * [new branch] gh/bobrenjc93/617/base -> origin/gh/bobrenjc93/617/base 2025-10-10T00:37:13.0226000Z * [new branch] gh/bobrenjc93/617/head -> origin/gh/bobrenjc93/617/head 2025-10-10T00:37:13.0226155Z * [new branch] gh/bobrenjc93/617/orig -> origin/gh/bobrenjc93/617/orig 2025-10-10T00:37:13.0226661Z * [new branch] gh/bobrenjc93/618/base -> origin/gh/bobrenjc93/618/base 2025-10-10T00:37:13.0226848Z * [new branch] gh/bobrenjc93/618/head -> origin/gh/bobrenjc93/618/head 2025-10-10T00:37:13.0227008Z * [new branch] gh/bobrenjc93/618/orig -> origin/gh/bobrenjc93/618/orig 2025-10-10T00:37:13.0227170Z * [new branch] gh/bobrenjc93/619/base -> origin/gh/bobrenjc93/619/base 2025-10-10T00:37:13.0227384Z * [new branch] gh/bobrenjc93/619/head -> origin/gh/bobrenjc93/619/head 2025-10-10T00:37:13.0227535Z * [new branch] gh/bobrenjc93/619/orig -> origin/gh/bobrenjc93/619/orig 2025-10-10T00:37:13.0227700Z * [new branch] gh/bobrenjc93/620/base -> origin/gh/bobrenjc93/620/base 2025-10-10T00:37:13.0227903Z * [new branch] gh/bobrenjc93/620/head -> origin/gh/bobrenjc93/620/head 2025-10-10T00:37:13.0228067Z * [new branch] gh/bobrenjc93/620/orig -> origin/gh/bobrenjc93/620/orig 2025-10-10T00:37:13.0228215Z * [new branch] gh/bobrenjc93/621/base -> origin/gh/bobrenjc93/621/base 2025-10-10T00:37:13.0228379Z * [new branch] gh/bobrenjc93/621/head -> origin/gh/bobrenjc93/621/head 2025-10-10T00:37:13.0228535Z * [new branch] gh/bobrenjc93/621/orig -> origin/gh/bobrenjc93/621/orig 2025-10-10T00:37:13.0228685Z * [new branch] gh/bobrenjc93/622/base -> origin/gh/bobrenjc93/622/base 2025-10-10T00:37:13.0233788Z * [new branch] gh/bobrenjc93/622/head -> origin/gh/bobrenjc93/622/head 2025-10-10T00:37:13.0238684Z * [new branch] gh/bobrenjc93/622/orig -> origin/gh/bobrenjc93/622/orig 2025-10-10T00:37:13.0242752Z * [new branch] gh/bobrenjc93/623/base -> origin/gh/bobrenjc93/623/base 2025-10-10T00:37:13.0244731Z * [new branch] gh/bobrenjc93/623/head -> origin/gh/bobrenjc93/623/head 2025-10-10T00:37:13.0245401Z * [new branch] gh/bobrenjc93/623/orig -> origin/gh/bobrenjc93/623/orig 2025-10-10T00:37:13.0245589Z * [new branch] gh/bobrenjc93/624/base -> origin/gh/bobrenjc93/624/base 2025-10-10T00:37:13.0245752Z * [new branch] gh/bobrenjc93/624/head -> origin/gh/bobrenjc93/624/head 2025-10-10T00:37:13.0245912Z * [new branch] gh/bobrenjc93/624/orig -> origin/gh/bobrenjc93/624/orig 2025-10-10T00:37:13.0246072Z * [new branch] gh/bobrenjc93/625/base -> origin/gh/bobrenjc93/625/base 2025-10-10T00:37:13.0246235Z * [new branch] gh/bobrenjc93/625/head -> origin/gh/bobrenjc93/625/head 2025-10-10T00:37:13.0246394Z * [new branch] gh/bobrenjc93/625/orig -> origin/gh/bobrenjc93/625/orig 2025-10-10T00:37:13.0246554Z * [new branch] gh/bobrenjc93/626/base -> origin/gh/bobrenjc93/626/base 2025-10-10T00:37:13.0246710Z * [new branch] gh/bobrenjc93/626/head -> origin/gh/bobrenjc93/626/head 2025-10-10T00:37:13.0246871Z * [new branch] gh/bobrenjc93/626/orig -> origin/gh/bobrenjc93/626/orig 2025-10-10T00:37:13.0247031Z * [new branch] gh/bobrenjc93/627/base -> origin/gh/bobrenjc93/627/base 2025-10-10T00:37:13.0247188Z * [new branch] gh/bobrenjc93/627/head -> origin/gh/bobrenjc93/627/head 2025-10-10T00:37:13.0247347Z * [new branch] gh/bobrenjc93/627/orig -> origin/gh/bobrenjc93/627/orig 2025-10-10T00:37:13.0247500Z * [new branch] gh/bobrenjc93/628/base -> origin/gh/bobrenjc93/628/base 2025-10-10T00:37:13.0247656Z * [new branch] gh/bobrenjc93/628/head -> origin/gh/bobrenjc93/628/head 2025-10-10T00:37:13.0247809Z * [new branch] gh/bobrenjc93/628/orig -> origin/gh/bobrenjc93/628/orig 2025-10-10T00:37:13.0247968Z * [new branch] gh/bobrenjc93/629/base -> origin/gh/bobrenjc93/629/base 2025-10-10T00:37:13.0248135Z * [new branch] gh/bobrenjc93/629/head -> origin/gh/bobrenjc93/629/head 2025-10-10T00:37:13.0249163Z * [new branch] gh/bobrenjc93/629/orig -> origin/gh/bobrenjc93/629/orig 2025-10-10T00:37:13.0249367Z * [new branch] gh/bobrenjc93/630/base -> origin/gh/bobrenjc93/630/base 2025-10-10T00:37:13.0249550Z * [new branch] gh/bobrenjc93/630/head -> origin/gh/bobrenjc93/630/head 2025-10-10T00:37:13.0249872Z * [new branch] gh/bobrenjc93/630/orig -> origin/gh/bobrenjc93/630/orig 2025-10-10T00:37:13.0250198Z * [new branch] gh/bobrenjc93/631/base -> origin/gh/bobrenjc93/631/base 2025-10-10T00:37:13.0250361Z * [new branch] gh/bobrenjc93/631/head -> origin/gh/bobrenjc93/631/head 2025-10-10T00:37:13.0254599Z * [new branch] gh/bobrenjc93/631/orig -> origin/gh/bobrenjc93/631/orig 2025-10-10T00:37:13.0257377Z * [new branch] gh/bobrenjc93/632/base -> origin/gh/bobrenjc93/632/base 2025-10-10T00:37:13.0257651Z * [new branch] gh/bobrenjc93/632/head -> origin/gh/bobrenjc93/632/head 2025-10-10T00:37:13.0258195Z * [new branch] gh/bobrenjc93/632/orig -> origin/gh/bobrenjc93/632/orig 2025-10-10T00:37:13.0258379Z * [new branch] gh/bobrenjc93/633/base -> origin/gh/bobrenjc93/633/base 2025-10-10T00:37:13.0258540Z * [new branch] gh/bobrenjc93/633/head -> origin/gh/bobrenjc93/633/head 2025-10-10T00:37:13.0258709Z * [new branch] gh/bobrenjc93/633/orig -> origin/gh/bobrenjc93/633/orig 2025-10-10T00:37:13.0258865Z * [new branch] gh/bobrenjc93/634/base -> origin/gh/bobrenjc93/634/base 2025-10-10T00:37:13.0259032Z * [new branch] gh/bobrenjc93/634/head -> origin/gh/bobrenjc93/634/head 2025-10-10T00:37:13.0260591Z * [new branch] gh/bobrenjc93/634/orig -> origin/gh/bobrenjc93/634/orig 2025-10-10T00:37:13.0260905Z * [new branch] gh/bobrenjc93/635/base -> origin/gh/bobrenjc93/635/base 2025-10-10T00:37:13.0261059Z * [new branch] gh/bobrenjc93/635/head -> origin/gh/bobrenjc93/635/head 2025-10-10T00:37:13.0261224Z * [new branch] gh/bobrenjc93/635/orig -> origin/gh/bobrenjc93/635/orig 2025-10-10T00:37:13.0261429Z * [new branch] gh/bobrenjc93/636/base -> origin/gh/bobrenjc93/636/base 2025-10-10T00:37:13.0261601Z * [new branch] gh/bobrenjc93/636/head -> origin/gh/bobrenjc93/636/head 2025-10-10T00:37:13.0263286Z * [new branch] gh/bobrenjc93/636/orig -> origin/gh/bobrenjc93/636/orig 2025-10-10T00:37:13.0263466Z * [new branch] gh/bobrenjc93/637/base -> origin/gh/bobrenjc93/637/base 2025-10-10T00:37:13.0263867Z * [new branch] gh/bobrenjc93/637/head -> origin/gh/bobrenjc93/637/head 2025-10-10T00:37:13.0264853Z * [new branch] gh/bobrenjc93/637/orig -> origin/gh/bobrenjc93/637/orig 2025-10-10T00:37:13.0265761Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-10-10T00:37:13.0266624Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-10-10T00:37:13.0267295Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-10-10T00:37:13.0268415Z * [new branch] gh/bobrenjc93/639/base -> origin/gh/bobrenjc93/639/base 2025-10-10T00:37:13.0268726Z * [new branch] gh/bobrenjc93/639/head -> origin/gh/bobrenjc93/639/head 2025-10-10T00:37:13.0269831Z * [new branch] gh/bobrenjc93/639/orig -> origin/gh/bobrenjc93/639/orig 2025-10-10T00:37:13.0270608Z * [new branch] gh/bobrenjc93/640/base -> origin/gh/bobrenjc93/640/base 2025-10-10T00:37:13.0271242Z * [new branch] gh/bobrenjc93/640/head -> origin/gh/bobrenjc93/640/head 2025-10-10T00:37:13.0272189Z * [new branch] gh/bobrenjc93/640/orig -> origin/gh/bobrenjc93/640/orig 2025-10-10T00:37:13.0273173Z * [new branch] gh/bobrenjc93/641/base -> origin/gh/bobrenjc93/641/base 2025-10-10T00:37:13.0273592Z * [new branch] gh/bobrenjc93/641/head -> origin/gh/bobrenjc93/641/head 2025-10-10T00:37:13.0274813Z * [new branch] gh/bobrenjc93/641/orig -> origin/gh/bobrenjc93/641/orig 2025-10-10T00:37:13.0279198Z * [new branch] gh/bobrenjc93/642/base -> origin/gh/bobrenjc93/642/base 2025-10-10T00:37:13.0279932Z * [new branch] gh/bobrenjc93/642/head -> origin/gh/bobrenjc93/642/head 2025-10-10T00:37:13.0280124Z * [new branch] gh/bobrenjc93/642/orig -> origin/gh/bobrenjc93/642/orig 2025-10-10T00:37:13.0280298Z * [new branch] gh/bobrenjc93/643/base -> origin/gh/bobrenjc93/643/base 2025-10-10T00:37:13.0280459Z * [new branch] gh/bobrenjc93/643/head -> origin/gh/bobrenjc93/643/head 2025-10-10T00:37:13.0280651Z * [new branch] gh/bobrenjc93/643/orig -> origin/gh/bobrenjc93/643/orig 2025-10-10T00:37:13.0282916Z * [new branch] gh/bobrenjc93/644/base -> origin/gh/bobrenjc93/644/base 2025-10-10T00:37:13.0283166Z * [new branch] gh/bobrenjc93/644/head -> origin/gh/bobrenjc93/644/head 2025-10-10T00:37:13.0283348Z * [new branch] gh/bobrenjc93/644/orig -> origin/gh/bobrenjc93/644/orig 2025-10-10T00:37:13.0283652Z * [new branch] gh/bobrenjc93/645/base -> origin/gh/bobrenjc93/645/base 2025-10-10T00:37:13.0283825Z * [new branch] gh/bobrenjc93/645/head -> origin/gh/bobrenjc93/645/head 2025-10-10T00:37:13.0284087Z * [new branch] gh/bobrenjc93/645/orig -> origin/gh/bobrenjc93/645/orig 2025-10-10T00:37:13.0287941Z * [new branch] gh/bobrenjc93/646/base -> origin/gh/bobrenjc93/646/base 2025-10-10T00:37:13.0291222Z * [new branch] gh/bobrenjc93/646/head -> origin/gh/bobrenjc93/646/head 2025-10-10T00:37:13.0291604Z * [new branch] gh/bobrenjc93/646/orig -> origin/gh/bobrenjc93/646/orig 2025-10-10T00:37:13.0291780Z * [new branch] gh/bobrenjc93/647/base -> origin/gh/bobrenjc93/647/base 2025-10-10T00:37:13.0292025Z * [new branch] gh/bobrenjc93/647/head -> origin/gh/bobrenjc93/647/head 2025-10-10T00:37:13.0292188Z * [new branch] gh/bobrenjc93/647/orig -> origin/gh/bobrenjc93/647/orig 2025-10-10T00:37:13.0292454Z * [new branch] gh/bobrenjc93/648/base -> origin/gh/bobrenjc93/648/base 2025-10-10T00:37:13.0293082Z * [new branch] gh/bobrenjc93/648/head -> origin/gh/bobrenjc93/648/head 2025-10-10T00:37:13.0293375Z * [new branch] gh/bobrenjc93/648/orig -> origin/gh/bobrenjc93/648/orig 2025-10-10T00:37:13.0293655Z * [new branch] gh/bobrenjc93/649/base -> origin/gh/bobrenjc93/649/base 2025-10-10T00:37:13.0301871Z * [new branch] gh/bobrenjc93/649/head -> origin/gh/bobrenjc93/649/head 2025-10-10T00:37:13.0304301Z * [new branch] gh/bobrenjc93/649/orig -> origin/gh/bobrenjc93/649/orig 2025-10-10T00:37:13.0304532Z * [new branch] gh/bobrenjc93/650/base -> origin/gh/bobrenjc93/650/base 2025-10-10T00:37:13.0304732Z * [new branch] gh/bobrenjc93/650/head -> origin/gh/bobrenjc93/650/head 2025-10-10T00:37:13.0304918Z * [new branch] gh/bobrenjc93/650/orig -> origin/gh/bobrenjc93/650/orig 2025-10-10T00:37:13.0305110Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-10-10T00:37:13.0305280Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-10-10T00:37:13.0305427Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-10-10T00:37:13.0305562Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-10-10T00:37:13.0305723Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-10-10T00:37:13.0305858Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-10-10T00:37:13.0305994Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-10-10T00:37:13.0306128Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-10-10T00:37:13.0306703Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-10-10T00:37:13.0306853Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-10-10T00:37:13.0306990Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-10-10T00:37:13.0307131Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-10-10T00:37:13.0314475Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-10-10T00:37:13.0320108Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-10-10T00:37:13.0322176Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-10-10T00:37:13.0322345Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-10-10T00:37:13.0322831Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-10-10T00:37:13.0323055Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-10-10T00:37:13.0323240Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-10-10T00:37:13.0323404Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-10-10T00:37:13.0323555Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-10-10T00:37:13.0323705Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-10-10T00:37:13.0323997Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-10-10T00:37:13.0324153Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-10-10T00:37:13.0324312Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-10-10T00:37:13.0324465Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-10-10T00:37:13.0324636Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-10-10T00:37:13.0324790Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-10-10T00:37:13.0324945Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-10-10T00:37:13.0329743Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-10-10T00:37:13.0334872Z * [new branch] gh/coconutruben/36/base -> origin/gh/coconutruben/36/base 2025-10-10T00:37:13.0336316Z * [new branch] gh/coconutruben/36/head -> origin/gh/coconutruben/36/head 2025-10-10T00:37:13.0336603Z * [new branch] gh/coconutruben/36/orig -> origin/gh/coconutruben/36/orig 2025-10-10T00:37:13.0336787Z * [new branch] gh/coconutruben/48/base -> origin/gh/coconutruben/48/base 2025-10-10T00:37:13.0337042Z * [new branch] gh/coconutruben/48/head -> origin/gh/coconutruben/48/head 2025-10-10T00:37:13.0337269Z * [new branch] gh/coconutruben/48/orig -> origin/gh/coconutruben/48/orig 2025-10-10T00:37:13.0337502Z * [new branch] gh/coconutruben/49/base -> origin/gh/coconutruben/49/base 2025-10-10T00:37:13.0338141Z * [new branch] gh/coconutruben/49/head -> origin/gh/coconutruben/49/head 2025-10-10T00:37:13.0338357Z * [new branch] gh/coconutruben/49/orig -> origin/gh/coconutruben/49/orig 2025-10-10T00:37:13.0338552Z * [new branch] gh/coconutruben/50/base -> origin/gh/coconutruben/50/base 2025-10-10T00:37:13.0338724Z * [new branch] gh/coconutruben/50/head -> origin/gh/coconutruben/50/head 2025-10-10T00:37:13.0338889Z * [new branch] gh/coconutruben/50/orig -> origin/gh/coconutruben/50/orig 2025-10-10T00:37:13.0339060Z * [new branch] gh/coconutruben/51/base -> origin/gh/coconutruben/51/base 2025-10-10T00:37:13.0339478Z * [new branch] gh/coconutruben/51/head -> origin/gh/coconutruben/51/head 2025-10-10T00:37:13.0339658Z * [new branch] gh/coconutruben/51/orig -> origin/gh/coconutruben/51/orig 2025-10-10T00:37:13.0340755Z * [new branch] gh/coconutruben/52/base -> origin/gh/coconutruben/52/base 2025-10-10T00:37:13.0341288Z * [new branch] gh/coconutruben/52/head -> origin/gh/coconutruben/52/head 2025-10-10T00:37:13.0342606Z * [new branch] gh/coconutruben/52/orig -> origin/gh/coconutruben/52/orig 2025-10-10T00:37:13.0343275Z * [new branch] gh/coconutruben/53/base -> origin/gh/coconutruben/53/base 2025-10-10T00:37:13.0345141Z * [new branch] gh/coconutruben/53/head -> origin/gh/coconutruben/53/head 2025-10-10T00:37:13.0345438Z * [new branch] gh/coconutruben/53/orig -> origin/gh/coconutruben/53/orig 2025-10-10T00:37:13.0345870Z * [new branch] gh/coconutruben/54/base -> origin/gh/coconutruben/54/base 2025-10-10T00:37:13.0347582Z * [new branch] gh/coconutruben/54/head -> origin/gh/coconutruben/54/head 2025-10-10T00:37:13.0347928Z * [new branch] gh/coconutruben/54/orig -> origin/gh/coconutruben/54/orig 2025-10-10T00:37:13.0351414Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-10-10T00:37:13.0351761Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-10-10T00:37:13.0352216Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-10-10T00:37:13.0352510Z * [new branch] gh/coconutruben/56/base -> origin/gh/coconutruben/56/base 2025-10-10T00:37:13.0353270Z * [new branch] gh/coconutruben/56/head -> origin/gh/coconutruben/56/head 2025-10-10T00:37:13.0353465Z * [new branch] gh/coconutruben/56/orig -> origin/gh/coconutruben/56/orig 2025-10-10T00:37:13.0354224Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-10-10T00:37:13.0355380Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-10-10T00:37:13.0355708Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-10-10T00:37:13.0359055Z * [new branch] gh/coconutruben/58/base -> origin/gh/coconutruben/58/base 2025-10-10T00:37:13.0359253Z * [new branch] gh/coconutruben/58/head -> origin/gh/coconutruben/58/head 2025-10-10T00:37:13.0359426Z * [new branch] gh/coconutruben/58/orig -> origin/gh/coconutruben/58/orig 2025-10-10T00:37:13.0359694Z * [new branch] gh/coconutruben/59/base -> origin/gh/coconutruben/59/base 2025-10-10T00:37:13.0364691Z * [new branch] gh/coconutruben/59/head -> origin/gh/coconutruben/59/head 2025-10-10T00:37:13.0365031Z * [new branch] gh/coconutruben/59/orig -> origin/gh/coconutruben/59/orig 2025-10-10T00:37:13.0365307Z * [new branch] gh/coconutruben/62/base -> origin/gh/coconutruben/62/base 2025-10-10T00:37:13.0365476Z * [new branch] gh/coconutruben/62/head -> origin/gh/coconutruben/62/head 2025-10-10T00:37:13.0365754Z * [new branch] gh/coconutruben/62/orig -> origin/gh/coconutruben/62/orig 2025-10-10T00:37:13.0365926Z * [new branch] gh/coconutruben/64/base -> origin/gh/coconutruben/64/base 2025-10-10T00:37:13.0371841Z * [new branch] gh/coconutruben/64/head -> origin/gh/coconutruben/64/head 2025-10-10T00:37:13.0372065Z * [new branch] gh/coconutruben/64/orig -> origin/gh/coconutruben/64/orig 2025-10-10T00:37:13.0372228Z * [new branch] gh/coconutruben/65/base -> origin/gh/coconutruben/65/base 2025-10-10T00:37:13.0372543Z * [new branch] gh/coconutruben/65/head -> origin/gh/coconutruben/65/head 2025-10-10T00:37:13.0372977Z * [new branch] gh/coconutruben/65/orig -> origin/gh/coconutruben/65/orig 2025-10-10T00:37:13.0373695Z * [new branch] gh/coconutruben/66/base -> origin/gh/coconutruben/66/base 2025-10-10T00:37:13.0374036Z * [new branch] gh/coconutruben/66/head -> origin/gh/coconutruben/66/head 2025-10-10T00:37:13.0374288Z * [new branch] gh/coconutruben/66/orig -> origin/gh/coconutruben/66/orig 2025-10-10T00:37:13.0374440Z * [new branch] gh/coconutruben/67/base -> origin/gh/coconutruben/67/base 2025-10-10T00:37:13.0374903Z * [new branch] gh/coconutruben/67/head -> origin/gh/coconutruben/67/head 2025-10-10T00:37:13.0375119Z * [new branch] gh/coconutruben/67/orig -> origin/gh/coconutruben/67/orig 2025-10-10T00:37:13.0378824Z * [new branch] gh/coconutruben/68/base -> origin/gh/coconutruben/68/base 2025-10-10T00:37:13.0379215Z * [new branch] gh/coconutruben/68/head -> origin/gh/coconutruben/68/head 2025-10-10T00:37:13.0379397Z * [new branch] gh/coconutruben/68/orig -> origin/gh/coconutruben/68/orig 2025-10-10T00:37:13.0379556Z * [new branch] gh/coconutruben/69/base -> origin/gh/coconutruben/69/base 2025-10-10T00:37:13.0379715Z * [new branch] gh/coconutruben/69/head -> origin/gh/coconutruben/69/head 2025-10-10T00:37:13.0379863Z * [new branch] gh/coconutruben/69/orig -> origin/gh/coconutruben/69/orig 2025-10-10T00:37:13.0380176Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-10-10T00:37:13.0380383Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-10-10T00:37:13.0381475Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-10-10T00:37:13.0384276Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-10-10T00:37:13.0384487Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-10-10T00:37:13.0384657Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-10-10T00:37:13.0384813Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-10-10T00:37:13.0385270Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-10-10T00:37:13.0386013Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-10-10T00:37:13.0386974Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-10-10T00:37:13.0389863Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-10-10T00:37:13.0394904Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-10-10T00:37:13.0397091Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-10-10T00:37:13.0397391Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-10-10T00:37:13.0400348Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-10-10T00:37:13.0400638Z * [new branch] gh/coconutruben/75/base -> origin/gh/coconutruben/75/base 2025-10-10T00:37:13.0405618Z * [new branch] gh/coconutruben/75/head -> origin/gh/coconutruben/75/head 2025-10-10T00:37:13.0408289Z * [new branch] gh/coconutruben/75/orig -> origin/gh/coconutruben/75/orig 2025-10-10T00:37:13.0408512Z * [new branch] gh/coconutruben/76/base -> origin/gh/coconutruben/76/base 2025-10-10T00:37:13.0408670Z * [new branch] gh/coconutruben/76/head -> origin/gh/coconutruben/76/head 2025-10-10T00:37:13.0408832Z * [new branch] gh/coconutruben/76/orig -> origin/gh/coconutruben/76/orig 2025-10-10T00:37:13.0409129Z * [new branch] gh/coconutruben/77/base -> origin/gh/coconutruben/77/base 2025-10-10T00:37:13.0409282Z * [new branch] gh/coconutruben/77/head -> origin/gh/coconutruben/77/head 2025-10-10T00:37:13.0409443Z * [new branch] gh/coconutruben/77/orig -> origin/gh/coconutruben/77/orig 2025-10-10T00:37:13.0409595Z * [new branch] gh/coconutruben/78/base -> origin/gh/coconutruben/78/base 2025-10-10T00:37:13.0409753Z * [new branch] gh/coconutruben/78/head -> origin/gh/coconutruben/78/head 2025-10-10T00:37:13.0409912Z * [new branch] gh/coconutruben/78/orig -> origin/gh/coconutruben/78/orig 2025-10-10T00:37:13.0410065Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-10-10T00:37:13.0410208Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-10-10T00:37:13.0410350Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-10-10T00:37:13.0410503Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-10-10T00:37:13.0410643Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-10-10T00:37:13.0410792Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-10-10T00:37:13.0410935Z * [new branch] gh/coconutruben/81/base -> origin/gh/coconutruben/81/base 2025-10-10T00:37:13.0411130Z * [new branch] gh/coconutruben/81/head -> origin/gh/coconutruben/81/head 2025-10-10T00:37:13.0411273Z * [new branch] gh/coconutruben/81/orig -> origin/gh/coconutruben/81/orig 2025-10-10T00:37:13.0412610Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-10-10T00:37:13.0418250Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-10-10T00:37:13.0421783Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-10-10T00:37:13.0421979Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-10-10T00:37:13.0422156Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-10-10T00:37:13.0422321Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-10-10T00:37:13.0422513Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-10-10T00:37:13.0422689Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-10-10T00:37:13.0422846Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-10-10T00:37:13.0423005Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-10-10T00:37:13.0423151Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-10-10T00:37:13.0423324Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-10-10T00:37:13.0423470Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-10-10T00:37:13.0423614Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-10-10T00:37:13.0423812Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-10-10T00:37:13.0423987Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-10-10T00:37:13.0424160Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-10-10T00:37:13.0424316Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-10-10T00:37:13.0424512Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-10-10T00:37:13.0425215Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-10-10T00:37:13.0426535Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-10-10T00:37:13.0427100Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-10-10T00:37:13.0427903Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-10-10T00:37:13.0431744Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-10-10T00:37:13.0431982Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-10-10T00:37:13.0432134Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-10-10T00:37:13.0432288Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-10-10T00:37:13.0434619Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-10-10T00:37:13.0435026Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-10-10T00:37:13.0435284Z * [new branch] gh/davidberard98/401/base -> origin/gh/davidberard98/401/base 2025-10-10T00:37:13.0435730Z * [new branch] gh/davidberard98/401/head -> origin/gh/davidberard98/401/head 2025-10-10T00:37:13.0436718Z * [new branch] gh/davidberard98/401/orig -> origin/gh/davidberard98/401/orig 2025-10-10T00:37:13.0440014Z * [new branch] gh/davidberard98/405/base -> origin/gh/davidberard98/405/base 2025-10-10T00:37:13.0440373Z * [new branch] gh/davidberard98/405/head -> origin/gh/davidberard98/405/head 2025-10-10T00:37:13.0440621Z * [new branch] gh/davidberard98/405/orig -> origin/gh/davidberard98/405/orig 2025-10-10T00:37:13.0440875Z * [new branch] gh/davidberard98/410/base -> origin/gh/davidberard98/410/base 2025-10-10T00:37:13.0441083Z * [new branch] gh/davidberard98/410/head -> origin/gh/davidberard98/410/head 2025-10-10T00:37:13.0441228Z * [new branch] gh/davidberard98/410/orig -> origin/gh/davidberard98/410/orig 2025-10-10T00:37:13.0442768Z * [new branch] gh/davidberard98/411/base -> origin/gh/davidberard98/411/base 2025-10-10T00:37:13.0443124Z * [new branch] gh/davidberard98/411/head -> origin/gh/davidberard98/411/head 2025-10-10T00:37:13.0443388Z * [new branch] gh/davidberard98/411/orig -> origin/gh/davidberard98/411/orig 2025-10-10T00:37:13.0445325Z * [new branch] gh/davidberard98/412/base -> origin/gh/davidberard98/412/base 2025-10-10T00:37:13.0445657Z * [new branch] gh/davidberard98/412/head -> origin/gh/davidberard98/412/head 2025-10-10T00:37:13.0445893Z * [new branch] gh/davidberard98/412/orig -> origin/gh/davidberard98/412/orig 2025-10-10T00:37:13.0448695Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-10-10T00:37:13.0449038Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-10-10T00:37:13.0449261Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-10-10T00:37:13.0449437Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-10-10T00:37:13.0449612Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-10-10T00:37:13.0453622Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-10-10T00:37:13.0453951Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-10-10T00:37:13.0454172Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-10-10T00:37:13.0454392Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-10-10T00:37:13.0454843Z * [new branch] gh/desertfire/598/base -> origin/gh/desertfire/598/base 2025-10-10T00:37:13.0455039Z * [new branch] gh/desertfire/598/head -> origin/gh/desertfire/598/head 2025-10-10T00:37:13.0455205Z * [new branch] gh/desertfire/598/orig -> origin/gh/desertfire/598/orig 2025-10-10T00:37:13.0456271Z * [new branch] gh/desertfire/599/base -> origin/gh/desertfire/599/base 2025-10-10T00:37:13.0456529Z * [new branch] gh/desertfire/599/head -> origin/gh/desertfire/599/head 2025-10-10T00:37:13.0459549Z * [new branch] gh/desertfire/599/orig -> origin/gh/desertfire/599/orig 2025-10-10T00:37:13.0459874Z * [new branch] gh/desertfire/600/base -> origin/gh/desertfire/600/base 2025-10-10T00:37:13.0460043Z * [new branch] gh/desertfire/600/head -> origin/gh/desertfire/600/head 2025-10-10T00:37:13.0460289Z * [new branch] gh/desertfire/600/orig -> origin/gh/desertfire/600/orig 2025-10-10T00:37:13.0460455Z * [new branch] gh/desertfire/601/base -> origin/gh/desertfire/601/base 2025-10-10T00:37:13.0462351Z * [new branch] gh/desertfire/601/head -> origin/gh/desertfire/601/head 2025-10-10T00:37:13.0462519Z * [new branch] gh/desertfire/601/orig -> origin/gh/desertfire/601/orig 2025-10-10T00:37:13.0463346Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-10-10T00:37:13.0471433Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-10-10T00:37:13.0473221Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-10-10T00:37:13.0476394Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-10-10T00:37:13.0476648Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-10-10T00:37:13.0481578Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-10-10T00:37:13.0483874Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-10-10T00:37:13.0484051Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-10-10T00:37:13.0484184Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-10-10T00:37:13.0484319Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-10-10T00:37:13.0484461Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-10-10T00:37:13.0484591Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-10-10T00:37:13.0484727Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-10-10T00:37:13.0484854Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-10-10T00:37:13.0485010Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-10-10T00:37:13.0485143Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-10-10T00:37:13.0485278Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-10-10T00:37:13.0485404Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-10-10T00:37:13.0485537Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-10-10T00:37:13.0485668Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-10-10T00:37:13.0485795Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-10-10T00:37:13.0485929Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-10-10T00:37:13.0486054Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-10-10T00:37:13.0486320Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-10-10T00:37:13.0486448Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-10-10T00:37:13.0486576Z * [new branch] gh/drisspg/187/base -> origin/gh/drisspg/187/base 2025-10-10T00:37:13.0486713Z * [new branch] gh/drisspg/187/head -> origin/gh/drisspg/187/head 2025-10-10T00:37:13.0486840Z * [new branch] gh/drisspg/187/orig -> origin/gh/drisspg/187/orig 2025-10-10T00:37:13.0486985Z * [new branch] gh/drisspg/188/base -> origin/gh/drisspg/188/base 2025-10-10T00:37:13.0487109Z * [new branch] gh/drisspg/188/head -> origin/gh/drisspg/188/head 2025-10-10T00:37:13.0487240Z * [new branch] gh/drisspg/188/orig -> origin/gh/drisspg/188/orig 2025-10-10T00:37:13.0492668Z * [new branch] gh/drisspg/189/base -> origin/gh/drisspg/189/base 2025-10-10T00:37:13.0494394Z * [new branch] gh/drisspg/189/head -> origin/gh/drisspg/189/head 2025-10-10T00:37:13.0494668Z * [new branch] gh/drisspg/189/orig -> origin/gh/drisspg/189/orig 2025-10-10T00:37:13.0498360Z * [new branch] gh/drisspg/193/base -> origin/gh/drisspg/193/base 2025-10-10T00:37:13.0498679Z * [new branch] gh/drisspg/193/head -> origin/gh/drisspg/193/head 2025-10-10T00:37:13.0498855Z * [new branch] gh/drisspg/193/orig -> origin/gh/drisspg/193/orig 2025-10-10T00:37:13.0499248Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-10-10T00:37:13.0499428Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-10-10T00:37:13.0499586Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-10-10T00:37:13.0499743Z * [new branch] gh/drisspg/196/base -> origin/gh/drisspg/196/base 2025-10-10T00:37:13.0499911Z * [new branch] gh/drisspg/196/head -> origin/gh/drisspg/196/head 2025-10-10T00:37:13.0500063Z * [new branch] gh/drisspg/196/orig -> origin/gh/drisspg/196/orig 2025-10-10T00:37:13.0500228Z * [new branch] gh/drisspg/197/base -> origin/gh/drisspg/197/base 2025-10-10T00:37:13.0500378Z * [new branch] gh/drisspg/197/head -> origin/gh/drisspg/197/head 2025-10-10T00:37:13.0500528Z * [new branch] gh/drisspg/197/orig -> origin/gh/drisspg/197/orig 2025-10-10T00:37:13.0500683Z * [new branch] gh/drisspg/198/base -> origin/gh/drisspg/198/base 2025-10-10T00:37:13.0500823Z * [new branch] gh/drisspg/198/head -> origin/gh/drisspg/198/head 2025-10-10T00:37:13.0500977Z * [new branch] gh/drisspg/198/orig -> origin/gh/drisspg/198/orig 2025-10-10T00:37:13.0501125Z * [new branch] gh/drisspg/199/base -> origin/gh/drisspg/199/base 2025-10-10T00:37:13.0501270Z * [new branch] gh/drisspg/199/head -> origin/gh/drisspg/199/head 2025-10-10T00:37:13.0501415Z * [new branch] gh/drisspg/199/orig -> origin/gh/drisspg/199/orig 2025-10-10T00:37:13.0501561Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-10-10T00:37:13.0501711Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-10-10T00:37:13.0501865Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-10-10T00:37:13.0502021Z * [new branch] gh/drisspg/201/base -> origin/gh/drisspg/201/base 2025-10-10T00:37:13.0502163Z * [new branch] gh/drisspg/201/head -> origin/gh/drisspg/201/head 2025-10-10T00:37:13.0503110Z * [new branch] gh/drisspg/201/orig -> origin/gh/drisspg/201/orig 2025-10-10T00:37:13.0508003Z * [new branch] gh/drisspg/202/base -> origin/gh/drisspg/202/base 2025-10-10T00:37:13.0512869Z * [new branch] gh/drisspg/202/head -> origin/gh/drisspg/202/head 2025-10-10T00:37:13.0514988Z * [new branch] gh/drisspg/202/orig -> origin/gh/drisspg/202/orig 2025-10-10T00:37:13.0515166Z * [new branch] gh/drisspg/203/base -> origin/gh/drisspg/203/base 2025-10-10T00:37:13.0515318Z * [new branch] gh/drisspg/203/head -> origin/gh/drisspg/203/head 2025-10-10T00:37:13.0515479Z * [new branch] gh/drisspg/203/orig -> origin/gh/drisspg/203/orig 2025-10-10T00:37:13.0515630Z * [new branch] gh/drisspg/204/base -> origin/gh/drisspg/204/base 2025-10-10T00:37:13.0515776Z * [new branch] gh/drisspg/204/head -> origin/gh/drisspg/204/head 2025-10-10T00:37:13.0515923Z * [new branch] gh/drisspg/204/orig -> origin/gh/drisspg/204/orig 2025-10-10T00:37:13.0516103Z * [new branch] gh/drisspg/205/base -> origin/gh/drisspg/205/base 2025-10-10T00:37:13.0516248Z * [new branch] gh/drisspg/205/head -> origin/gh/drisspg/205/head 2025-10-10T00:37:13.0516399Z * [new branch] gh/drisspg/205/orig -> origin/gh/drisspg/205/orig 2025-10-10T00:37:13.0516547Z * [new branch] gh/drisspg/206/base -> origin/gh/drisspg/206/base 2025-10-10T00:37:13.0516700Z * [new branch] gh/drisspg/206/head -> origin/gh/drisspg/206/head 2025-10-10T00:37:13.0516989Z * [new branch] gh/drisspg/206/orig -> origin/gh/drisspg/206/orig 2025-10-10T00:37:13.0517142Z * [new branch] gh/drisspg/207/base -> origin/gh/drisspg/207/base 2025-10-10T00:37:13.0517298Z * [new branch] gh/drisspg/207/head -> origin/gh/drisspg/207/head 2025-10-10T00:37:13.0520032Z * [new branch] gh/drisspg/207/orig -> origin/gh/drisspg/207/orig 2025-10-10T00:37:13.0520233Z * [new branch] gh/drisspg/208/base -> origin/gh/drisspg/208/base 2025-10-10T00:37:13.0520416Z * [new branch] gh/drisspg/208/head -> origin/gh/drisspg/208/head 2025-10-10T00:37:13.0520598Z * [new branch] gh/drisspg/208/orig -> origin/gh/drisspg/208/orig 2025-10-10T00:37:13.0520755Z * [new branch] gh/drisspg/209/base -> origin/gh/drisspg/209/base 2025-10-10T00:37:13.0520901Z * [new branch] gh/drisspg/209/head -> origin/gh/drisspg/209/head 2025-10-10T00:37:13.0521109Z * [new branch] gh/drisspg/209/orig -> origin/gh/drisspg/209/orig 2025-10-10T00:37:13.0524491Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-10-10T00:37:13.0528376Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-10-10T00:37:13.0531632Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-10-10T00:37:13.0531968Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-10-10T00:37:13.0537717Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-10-10T00:37:13.0539912Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-10-10T00:37:13.0540106Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-10-10T00:37:13.0540256Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-10-10T00:37:13.0540416Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-10-10T00:37:13.0540564Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-10-10T00:37:13.0540704Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-10-10T00:37:13.0540851Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-10-10T00:37:13.0541186Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-10-10T00:37:13.0541346Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-10-10T00:37:13.0541490Z * [new branch] gh/eellison/824/base -> origin/gh/eellison/824/base 2025-10-10T00:37:13.0541629Z * [new branch] gh/eellison/824/head -> origin/gh/eellison/824/head 2025-10-10T00:37:13.0541776Z * [new branch] gh/eellison/824/orig -> origin/gh/eellison/824/orig 2025-10-10T00:37:13.0541921Z * [new branch] gh/eellison/825/base -> origin/gh/eellison/825/base 2025-10-10T00:37:13.0542083Z * [new branch] gh/eellison/825/head -> origin/gh/eellison/825/head 2025-10-10T00:37:13.0542227Z * [new branch] gh/eellison/825/orig -> origin/gh/eellison/825/orig 2025-10-10T00:37:13.0542365Z * [new branch] gh/eellison/826/base -> origin/gh/eellison/826/base 2025-10-10T00:37:13.0542514Z * [new branch] gh/eellison/826/head -> origin/gh/eellison/826/head 2025-10-10T00:37:13.0542657Z * [new branch] gh/eellison/826/orig -> origin/gh/eellison/826/orig 2025-10-10T00:37:13.0543088Z * [new branch] gh/eellison/827/base -> origin/gh/eellison/827/base 2025-10-10T00:37:13.0544005Z * [new branch] gh/eellison/827/head -> origin/gh/eellison/827/head 2025-10-10T00:37:13.0544459Z * [new branch] gh/eellison/827/orig -> origin/gh/eellison/827/orig 2025-10-10T00:37:13.0545693Z * [new branch] gh/eellison/828/base -> origin/gh/eellison/828/base 2025-10-10T00:37:13.0546107Z * [new branch] gh/eellison/828/head -> origin/gh/eellison/828/head 2025-10-10T00:37:13.0547214Z * [new branch] gh/eellison/828/orig -> origin/gh/eellison/828/orig 2025-10-10T00:37:13.0550679Z * [new branch] gh/eellison/829/base -> origin/gh/eellison/829/base 2025-10-10T00:37:13.0550870Z * [new branch] gh/eellison/829/head -> origin/gh/eellison/829/head 2025-10-10T00:37:13.0551018Z * [new branch] gh/eellison/829/orig -> origin/gh/eellison/829/orig 2025-10-10T00:37:13.0551193Z * [new branch] gh/eellison/830/base -> origin/gh/eellison/830/base 2025-10-10T00:37:13.0556694Z * [new branch] gh/eellison/830/head -> origin/gh/eellison/830/head 2025-10-10T00:37:13.0561603Z * [new branch] gh/eellison/830/orig -> origin/gh/eellison/830/orig 2025-10-10T00:37:13.0566405Z * [new branch] gh/eellison/831/base -> origin/gh/eellison/831/base 2025-10-10T00:37:13.0572098Z * [new branch] gh/eellison/831/head -> origin/gh/eellison/831/head 2025-10-10T00:37:13.0577348Z * [new branch] gh/eellison/831/orig -> origin/gh/eellison/831/orig 2025-10-10T00:37:13.0579528Z * [new branch] gh/eellison/832/base -> origin/gh/eellison/832/base 2025-10-10T00:37:13.0579837Z * [new branch] gh/eellison/832/head -> origin/gh/eellison/832/head 2025-10-10T00:37:13.0580007Z * [new branch] gh/eellison/832/orig -> origin/gh/eellison/832/orig 2025-10-10T00:37:13.0580237Z * [new branch] gh/eellison/833/base -> origin/gh/eellison/833/base 2025-10-10T00:37:13.0580393Z * [new branch] gh/eellison/833/head -> origin/gh/eellison/833/head 2025-10-10T00:37:13.0580636Z * [new branch] gh/eellison/833/orig -> origin/gh/eellison/833/orig 2025-10-10T00:37:13.0580792Z * [new branch] gh/eellison/834/base -> origin/gh/eellison/834/base 2025-10-10T00:37:13.0581014Z * [new branch] gh/eellison/834/head -> origin/gh/eellison/834/head 2025-10-10T00:37:13.0581729Z * [new branch] gh/eellison/834/orig -> origin/gh/eellison/834/orig 2025-10-10T00:37:13.0582102Z * [new branch] gh/eellison/835/base -> origin/gh/eellison/835/base 2025-10-10T00:37:13.0582256Z * [new branch] gh/eellison/835/head -> origin/gh/eellison/835/head 2025-10-10T00:37:13.0582417Z * [new branch] gh/eellison/835/orig -> origin/gh/eellison/835/orig 2025-10-10T00:37:13.0582567Z * [new branch] gh/eellison/836/base -> origin/gh/eellison/836/base 2025-10-10T00:37:13.0582720Z * [new branch] gh/eellison/836/head -> origin/gh/eellison/836/head 2025-10-10T00:37:13.0582891Z * [new branch] gh/eellison/836/orig -> origin/gh/eellison/836/orig 2025-10-10T00:37:13.0583044Z * [new branch] gh/eellison/837/base -> origin/gh/eellison/837/base 2025-10-10T00:37:13.0583197Z * [new branch] gh/eellison/837/head -> origin/gh/eellison/837/head 2025-10-10T00:37:13.0583348Z * [new branch] gh/eellison/837/orig -> origin/gh/eellison/837/orig 2025-10-10T00:37:13.0583510Z * [new branch] gh/eellison/838/base -> origin/gh/eellison/838/base 2025-10-10T00:37:13.0583659Z * [new branch] gh/eellison/838/head -> origin/gh/eellison/838/head 2025-10-10T00:37:13.0583807Z * [new branch] gh/eellison/838/orig -> origin/gh/eellison/838/orig 2025-10-10T00:37:13.0583963Z * [new branch] gh/eellison/839/base -> origin/gh/eellison/839/base 2025-10-10T00:37:13.0584159Z * [new branch] gh/eellison/839/head -> origin/gh/eellison/839/head 2025-10-10T00:37:13.0584316Z * [new branch] gh/eellison/839/orig -> origin/gh/eellison/839/orig 2025-10-10T00:37:13.0584465Z * [new branch] gh/eellison/840/base -> origin/gh/eellison/840/base 2025-10-10T00:37:13.0584615Z * [new branch] gh/eellison/840/head -> origin/gh/eellison/840/head 2025-10-10T00:37:13.0584786Z * [new branch] gh/eellison/840/orig -> origin/gh/eellison/840/orig 2025-10-10T00:37:13.0584936Z * [new branch] gh/eellison/841/base -> origin/gh/eellison/841/base 2025-10-10T00:37:13.0585094Z * [new branch] gh/eellison/841/head -> origin/gh/eellison/841/head 2025-10-10T00:37:13.0585242Z * [new branch] gh/eellison/841/orig -> origin/gh/eellison/841/orig 2025-10-10T00:37:13.0585399Z * [new branch] gh/eellison/842/base -> origin/gh/eellison/842/base 2025-10-10T00:37:13.0585549Z * [new branch] gh/eellison/842/head -> origin/gh/eellison/842/head 2025-10-10T00:37:13.0585698Z * [new branch] gh/eellison/842/orig -> origin/gh/eellison/842/orig 2025-10-10T00:37:13.0585859Z * [new branch] gh/eellison/843/base -> origin/gh/eellison/843/base 2025-10-10T00:37:13.0586009Z * [new branch] gh/eellison/843/head -> origin/gh/eellison/843/head 2025-10-10T00:37:13.0586161Z * [new branch] gh/eellison/843/orig -> origin/gh/eellison/843/orig 2025-10-10T00:37:13.0586598Z * [new branch] gh/eellison/844/base -> origin/gh/eellison/844/base 2025-10-10T00:37:13.0586764Z * [new branch] gh/eellison/844/head -> origin/gh/eellison/844/head 2025-10-10T00:37:13.0586912Z * [new branch] gh/eellison/844/orig -> origin/gh/eellison/844/orig 2025-10-10T00:37:13.0587553Z * [new branch] gh/eellison/845/base -> origin/gh/eellison/845/base 2025-10-10T00:37:13.0594156Z * [new branch] gh/eellison/845/head -> origin/gh/eellison/845/head 2025-10-10T00:37:13.0603364Z * [new branch] gh/eellison/845/orig -> origin/gh/eellison/845/orig 2025-10-10T00:37:13.0603592Z * [new branch] gh/eellison/846/base -> origin/gh/eellison/846/base 2025-10-10T00:37:13.0603752Z * [new branch] gh/eellison/846/head -> origin/gh/eellison/846/head 2025-10-10T00:37:13.0604075Z * [new branch] gh/eellison/846/orig -> origin/gh/eellison/846/orig 2025-10-10T00:37:13.0604224Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-10-10T00:37:13.0604446Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-10-10T00:37:13.0604638Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-10-10T00:37:13.0604778Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-10-10T00:37:13.0604933Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-10-10T00:37:13.0605097Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-10-10T00:37:13.0605236Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-10-10T00:37:13.0605370Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-10-10T00:37:13.0605514Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-10-10T00:37:13.0605648Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-10-10T00:37:13.0605780Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-10-10T00:37:13.0605925Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-10-10T00:37:13.0606056Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-10-10T00:37:13.0606237Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-10-10T00:37:13.0606370Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-10-10T00:37:13.0606753Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-10-10T00:37:13.0606889Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-10-10T00:37:13.0607033Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-10-10T00:37:13.0607175Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-10-10T00:37:13.0607456Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-10-10T00:37:13.0607609Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-10-10T00:37:13.0609594Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-10-10T00:37:13.0609945Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-10-10T00:37:13.0616571Z * [new branch] gh/etaf/162/base -> origin/gh/etaf/162/base 2025-10-10T00:37:13.0616907Z * [new branch] gh/etaf/162/head -> origin/gh/etaf/162/head 2025-10-10T00:37:13.0617077Z * [new branch] gh/etaf/162/orig -> origin/gh/etaf/162/orig 2025-10-10T00:37:13.0617307Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-10-10T00:37:13.0617552Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-10-10T00:37:13.0617729Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-10-10T00:37:13.0618314Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-10-10T00:37:13.0620299Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-10-10T00:37:13.0620501Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-10-10T00:37:13.0620651Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-10-10T00:37:13.0620784Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-10-10T00:37:13.0620926Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-10-10T00:37:13.0621229Z * [new branch] gh/etaf/170/base -> origin/gh/etaf/170/base 2025-10-10T00:37:13.0621367Z * [new branch] gh/etaf/170/head -> origin/gh/etaf/170/head 2025-10-10T00:37:13.0621517Z * [new branch] gh/etaf/170/orig -> origin/gh/etaf/170/orig 2025-10-10T00:37:13.0621663Z * [new branch] gh/etaf/171/base -> origin/gh/etaf/171/base 2025-10-10T00:37:13.0621808Z * [new branch] gh/etaf/171/head -> origin/gh/etaf/171/head 2025-10-10T00:37:13.0622519Z * [new branch] gh/etaf/171/orig -> origin/gh/etaf/171/orig 2025-10-10T00:37:13.0623936Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-10-10T00:37:13.0628382Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-10-10T00:37:13.0632198Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-10-10T00:37:13.0635748Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-10-10T00:37:13.0636061Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-10-10T00:37:13.0636321Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-10-10T00:37:13.0636524Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-10-10T00:37:13.0636764Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-10-10T00:37:13.0637296Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-10-10T00:37:13.0637445Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-10-10T00:37:13.0637598Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-10-10T00:37:13.0637744Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-10-10T00:37:13.0637888Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-10-10T00:37:13.0638046Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-10-10T00:37:13.0638230Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-10-10T00:37:13.0641710Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-10-10T00:37:13.0646055Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-10-10T00:37:13.0652313Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-10-10T00:37:13.0657291Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-10-10T00:37:13.0662432Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-10-10T00:37:13.0663917Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-10-10T00:37:13.0664100Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-10-10T00:37:13.0664239Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-10-10T00:37:13.0664372Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-10-10T00:37:13.0664511Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-10-10T00:37:13.0664649Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-10-10T00:37:13.0664785Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-10-10T00:37:13.0664914Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-10-10T00:37:13.0665050Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-10-10T00:37:13.0665440Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-10-10T00:37:13.0665571Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-10-10T00:37:13.0665713Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-10-10T00:37:13.0665853Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-10-10T00:37:13.0665989Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-10-10T00:37:13.0666122Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-10-10T00:37:13.0666436Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-10-10T00:37:13.0666579Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-10-10T00:37:13.0666708Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-10-10T00:37:13.0666848Z * [new branch] gh/ezyang/3138/base -> origin/gh/ezyang/3138/base 2025-10-10T00:37:13.0666981Z * [new branch] gh/ezyang/3138/head -> origin/gh/ezyang/3138/head 2025-10-10T00:37:13.0667129Z * [new branch] gh/ezyang/3138/orig -> origin/gh/ezyang/3138/orig 2025-10-10T00:37:13.0667270Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-10-10T00:37:13.0667418Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-10-10T00:37:13.0667607Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-10-10T00:37:13.0667749Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-10-10T00:37:13.0667896Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-10-10T00:37:13.0668038Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-10-10T00:37:13.0668189Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-10-10T00:37:13.0668495Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-10-10T00:37:13.0668750Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-10-10T00:37:13.0668998Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-10-10T00:37:13.0669132Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-10-10T00:37:13.0669272Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-10-10T00:37:13.0669399Z * [new branch] gh/ezyang/3145/base -> origin/gh/ezyang/3145/base 2025-10-10T00:37:13.0669531Z * [new branch] gh/ezyang/3145/head -> origin/gh/ezyang/3145/head 2025-10-10T00:37:13.0669667Z * [new branch] gh/ezyang/3145/orig -> origin/gh/ezyang/3145/orig 2025-10-10T00:37:13.0669794Z * [new branch] gh/ezyang/3146/base -> origin/gh/ezyang/3146/base 2025-10-10T00:37:13.0706167Z * [new branch] gh/ezyang/3146/head -> origin/gh/ezyang/3146/head 2025-10-10T00:37:13.0706525Z * [new branch] gh/ezyang/3146/orig -> origin/gh/ezyang/3146/orig 2025-10-10T00:37:13.0706705Z * [new branch] gh/ezyang/3147/base -> origin/gh/ezyang/3147/base 2025-10-10T00:37:13.0707338Z * [new branch] gh/ezyang/3147/head -> origin/gh/ezyang/3147/head 2025-10-10T00:37:13.0707566Z * [new branch] gh/ezyang/3147/orig -> origin/gh/ezyang/3147/orig 2025-10-10T00:37:13.0707720Z * [new branch] gh/ezyang/3148/base -> origin/gh/ezyang/3148/base 2025-10-10T00:37:13.0707869Z * [new branch] gh/ezyang/3148/head -> origin/gh/ezyang/3148/head 2025-10-10T00:37:13.0708264Z * [new branch] gh/ezyang/3148/orig -> origin/gh/ezyang/3148/orig 2025-10-10T00:37:13.0708674Z * [new branch] gh/ezyang/3149/base -> origin/gh/ezyang/3149/base 2025-10-10T00:37:13.0708849Z * [new branch] gh/ezyang/3149/head -> origin/gh/ezyang/3149/head 2025-10-10T00:37:13.0708993Z * [new branch] gh/ezyang/3149/orig -> origin/gh/ezyang/3149/orig 2025-10-10T00:37:13.0709149Z * [new branch] gh/ezyang/3150/base -> origin/gh/ezyang/3150/base 2025-10-10T00:37:13.0709300Z * [new branch] gh/ezyang/3150/head -> origin/gh/ezyang/3150/head 2025-10-10T00:37:13.0709447Z * [new branch] gh/ezyang/3150/orig -> origin/gh/ezyang/3150/orig 2025-10-10T00:37:13.0709599Z * [new branch] gh/ezyang/3151/base -> origin/gh/ezyang/3151/base 2025-10-10T00:37:13.0709744Z * [new branch] gh/ezyang/3151/head -> origin/gh/ezyang/3151/head 2025-10-10T00:37:13.0709898Z * [new branch] gh/ezyang/3151/orig -> origin/gh/ezyang/3151/orig 2025-10-10T00:37:13.0710046Z * [new branch] gh/ezyang/3152/base -> origin/gh/ezyang/3152/base 2025-10-10T00:37:13.0710190Z * [new branch] gh/ezyang/3152/head -> origin/gh/ezyang/3152/head 2025-10-10T00:37:13.0710336Z * [new branch] gh/ezyang/3152/orig -> origin/gh/ezyang/3152/orig 2025-10-10T00:37:13.0710482Z * [new branch] gh/ezyang/3153/base -> origin/gh/ezyang/3153/base 2025-10-10T00:37:13.0710878Z * [new branch] gh/ezyang/3153/head -> origin/gh/ezyang/3153/head 2025-10-10T00:37:13.0711030Z * [new branch] gh/ezyang/3153/orig -> origin/gh/ezyang/3153/orig 2025-10-10T00:37:13.0711182Z * [new branch] gh/ezyang/3154/base -> origin/gh/ezyang/3154/base 2025-10-10T00:37:13.0711330Z * [new branch] gh/ezyang/3154/head -> origin/gh/ezyang/3154/head 2025-10-10T00:37:13.0711480Z * [new branch] gh/ezyang/3154/orig -> origin/gh/ezyang/3154/orig 2025-10-10T00:37:13.0711640Z * [new branch] gh/ezyang/3155/base -> origin/gh/ezyang/3155/base 2025-10-10T00:37:13.0711788Z * [new branch] gh/ezyang/3155/head -> origin/gh/ezyang/3155/head 2025-10-10T00:37:13.0711942Z * [new branch] gh/ezyang/3155/orig -> origin/gh/ezyang/3155/orig 2025-10-10T00:37:13.0712089Z * [new branch] gh/ezyang/3156/base -> origin/gh/ezyang/3156/base 2025-10-10T00:37:13.0712246Z * [new branch] gh/ezyang/3156/head -> origin/gh/ezyang/3156/head 2025-10-10T00:37:13.0712395Z * [new branch] gh/ezyang/3156/orig -> origin/gh/ezyang/3156/orig 2025-10-10T00:37:13.0712540Z * [new branch] gh/ezyang/3157/base -> origin/gh/ezyang/3157/base 2025-10-10T00:37:13.0712687Z * [new branch] gh/ezyang/3157/head -> origin/gh/ezyang/3157/head 2025-10-10T00:37:13.0712838Z * [new branch] gh/ezyang/3157/orig -> origin/gh/ezyang/3157/orig 2025-10-10T00:37:13.0712992Z * [new branch] gh/ezyang/3158/base -> origin/gh/ezyang/3158/base 2025-10-10T00:37:13.0713136Z * [new branch] gh/ezyang/3158/head -> origin/gh/ezyang/3158/head 2025-10-10T00:37:13.0713282Z * [new branch] gh/ezyang/3158/orig -> origin/gh/ezyang/3158/orig 2025-10-10T00:37:13.0713436Z * [new branch] gh/ezyang/3159/base -> origin/gh/ezyang/3159/base 2025-10-10T00:37:13.0713583Z * [new branch] gh/ezyang/3159/head -> origin/gh/ezyang/3159/head 2025-10-10T00:37:13.0713731Z * [new branch] gh/ezyang/3159/orig -> origin/gh/ezyang/3159/orig 2025-10-10T00:37:13.0713878Z * [new branch] gh/ezyang/3160/base -> origin/gh/ezyang/3160/base 2025-10-10T00:37:13.0714029Z * [new branch] gh/ezyang/3160/head -> origin/gh/ezyang/3160/head 2025-10-10T00:37:13.0714202Z * [new branch] gh/ezyang/3160/orig -> origin/gh/ezyang/3160/orig 2025-10-10T00:37:13.0714347Z * [new branch] gh/ezyang/3161/base -> origin/gh/ezyang/3161/base 2025-10-10T00:37:13.0714489Z * [new branch] gh/ezyang/3161/head -> origin/gh/ezyang/3161/head 2025-10-10T00:37:13.0714623Z * [new branch] gh/ezyang/3161/orig -> origin/gh/ezyang/3161/orig 2025-10-10T00:37:13.0714772Z * [new branch] gh/ezyang/3162/base -> origin/gh/ezyang/3162/base 2025-10-10T00:37:13.0714924Z * [new branch] gh/ezyang/3162/head -> origin/gh/ezyang/3162/head 2025-10-10T00:37:13.0715058Z * [new branch] gh/ezyang/3162/orig -> origin/gh/ezyang/3162/orig 2025-10-10T00:37:13.0715216Z * [new branch] gh/ezyang/3163/base -> origin/gh/ezyang/3163/base 2025-10-10T00:37:13.0715364Z * [new branch] gh/ezyang/3163/head -> origin/gh/ezyang/3163/head 2025-10-10T00:37:13.0715513Z * [new branch] gh/ezyang/3163/orig -> origin/gh/ezyang/3163/orig 2025-10-10T00:37:13.0715661Z * [new branch] gh/ezyang/3164/base -> origin/gh/ezyang/3164/base 2025-10-10T00:37:13.0715809Z * [new branch] gh/ezyang/3164/head -> origin/gh/ezyang/3164/head 2025-10-10T00:37:13.0715962Z * [new branch] gh/ezyang/3164/orig -> origin/gh/ezyang/3164/orig 2025-10-10T00:37:13.0716102Z * [new branch] gh/ezyang/3165/base -> origin/gh/ezyang/3165/base 2025-10-10T00:37:13.0716289Z * [new branch] gh/ezyang/3165/head -> origin/gh/ezyang/3165/head 2025-10-10T00:37:13.0716430Z * [new branch] gh/ezyang/3165/orig -> origin/gh/ezyang/3165/orig 2025-10-10T00:37:13.0716584Z * [new branch] gh/ezyang/3166/base -> origin/gh/ezyang/3166/base 2025-10-10T00:37:13.0716737Z * [new branch] gh/ezyang/3166/head -> origin/gh/ezyang/3166/head 2025-10-10T00:37:13.0717995Z * [new branch] gh/ezyang/3166/orig -> origin/gh/ezyang/3166/orig 2025-10-10T00:37:13.0718633Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-10-10T00:37:13.0721569Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-10-10T00:37:13.0726378Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-10-10T00:37:13.0726703Z * [new branch] gh/ezyang/3168/base -> origin/gh/ezyang/3168/base 2025-10-10T00:37:13.0726882Z * [new branch] gh/ezyang/3168/head -> origin/gh/ezyang/3168/head 2025-10-10T00:37:13.0727029Z * [new branch] gh/ezyang/3168/orig -> origin/gh/ezyang/3168/orig 2025-10-10T00:37:13.0727197Z * [new branch] gh/ezyang/3169/base -> origin/gh/ezyang/3169/base 2025-10-10T00:37:13.0727348Z * [new branch] gh/ezyang/3169/head -> origin/gh/ezyang/3169/head 2025-10-10T00:37:13.0727497Z * [new branch] gh/ezyang/3169/orig -> origin/gh/ezyang/3169/orig 2025-10-10T00:37:13.0727648Z * [new branch] gh/ezyang/3170/base -> origin/gh/ezyang/3170/base 2025-10-10T00:37:13.0727802Z * [new branch] gh/ezyang/3170/head -> origin/gh/ezyang/3170/head 2025-10-10T00:37:13.0727946Z * [new branch] gh/ezyang/3170/orig -> origin/gh/ezyang/3170/orig 2025-10-10T00:37:13.0728124Z * [new branch] gh/ezyang/3171/base -> origin/gh/ezyang/3171/base 2025-10-10T00:37:13.0728265Z * [new branch] gh/ezyang/3171/head -> origin/gh/ezyang/3171/head 2025-10-10T00:37:13.0728410Z * [new branch] gh/ezyang/3171/orig -> origin/gh/ezyang/3171/orig 2025-10-10T00:37:13.0733006Z * [new branch] gh/ezyang/3172/base -> origin/gh/ezyang/3172/base 2025-10-10T00:37:13.0738525Z * [new branch] gh/ezyang/3172/head -> origin/gh/ezyang/3172/head 2025-10-10T00:37:13.0740385Z * [new branch] gh/ezyang/3172/orig -> origin/gh/ezyang/3172/orig 2025-10-10T00:37:13.0740535Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-10-10T00:37:13.0740680Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-10-10T00:37:13.0740812Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-10-10T00:37:13.0740990Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-10-10T00:37:13.0741123Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-10-10T00:37:13.0741252Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-10-10T00:37:13.0741391Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-10-10T00:37:13.0741525Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-10-10T00:37:13.0741745Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-10-10T00:37:13.0741876Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-10-10T00:37:13.0742021Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-10-10T00:37:13.0742149Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-10-10T00:37:13.0742458Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-10-10T00:37:13.0742604Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-10-10T00:37:13.0742734Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-10-10T00:37:13.0743019Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-10-10T00:37:13.0743160Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-10-10T00:37:13.0743298Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-10-10T00:37:13.0743486Z * [new branch] gh/fduwjj/183/base -> origin/gh/fduwjj/183/base 2025-10-10T00:37:13.0744555Z * [new branch] gh/fduwjj/183/head -> origin/gh/fduwjj/183/head 2025-10-10T00:37:13.0745173Z * [new branch] gh/fduwjj/183/orig -> origin/gh/fduwjj/183/orig 2025-10-10T00:37:13.0746619Z * [new branch] gh/fduwjj/184/base -> origin/gh/fduwjj/184/base 2025-10-10T00:37:13.0746993Z * [new branch] gh/fduwjj/184/head -> origin/gh/fduwjj/184/head 2025-10-10T00:37:13.0753401Z * [new branch] gh/fduwjj/184/orig -> origin/gh/fduwjj/184/orig 2025-10-10T00:37:13.0753579Z * [new branch] gh/fduwjj/185/base -> origin/gh/fduwjj/185/base 2025-10-10T00:37:13.0753739Z * [new branch] gh/fduwjj/185/head -> origin/gh/fduwjj/185/head 2025-10-10T00:37:13.0753915Z * [new branch] gh/fduwjj/185/orig -> origin/gh/fduwjj/185/orig 2025-10-10T00:37:13.0754061Z * [new branch] gh/fduwjj/191/base -> origin/gh/fduwjj/191/base 2025-10-10T00:37:13.0754208Z * [new branch] gh/fduwjj/191/head -> origin/gh/fduwjj/191/head 2025-10-10T00:37:13.0758842Z * [new branch] gh/fduwjj/191/orig -> origin/gh/fduwjj/191/orig 2025-10-10T00:37:13.0759029Z * [new branch] gh/fduwjj/192/base -> origin/gh/fduwjj/192/base 2025-10-10T00:37:13.0759189Z * [new branch] gh/fduwjj/192/head -> origin/gh/fduwjj/192/head 2025-10-10T00:37:13.0759342Z * [new branch] gh/fduwjj/192/orig -> origin/gh/fduwjj/192/orig 2025-10-10T00:37:13.0759495Z * [new branch] gh/fduwjj/193/base -> origin/gh/fduwjj/193/base 2025-10-10T00:37:13.0759796Z * [new branch] gh/fduwjj/193/head -> origin/gh/fduwjj/193/head 2025-10-10T00:37:13.0759958Z * [new branch] gh/fduwjj/193/orig -> origin/gh/fduwjj/193/orig 2025-10-10T00:37:13.0760109Z * [new branch] gh/fduwjj/194/base -> origin/gh/fduwjj/194/base 2025-10-10T00:37:13.0765097Z * [new branch] gh/fduwjj/194/head -> origin/gh/fduwjj/194/head 2025-10-10T00:37:13.0765276Z * [new branch] gh/fduwjj/194/orig -> origin/gh/fduwjj/194/orig 2025-10-10T00:37:13.0765428Z * [new branch] gh/fduwjj/195/base -> origin/gh/fduwjj/195/base 2025-10-10T00:37:13.0765554Z * [new branch] gh/fduwjj/195/head -> origin/gh/fduwjj/195/head 2025-10-10T00:37:13.0765689Z * [new branch] gh/fduwjj/195/orig -> origin/gh/fduwjj/195/orig 2025-10-10T00:37:13.0765817Z * [new branch] gh/fduwjj/196/base -> origin/gh/fduwjj/196/base 2025-10-10T00:37:13.0765961Z * [new branch] gh/fduwjj/196/head -> origin/gh/fduwjj/196/head 2025-10-10T00:37:13.0766088Z * [new branch] gh/fduwjj/196/orig -> origin/gh/fduwjj/196/orig 2025-10-10T00:37:13.0766553Z * [new branch] gh/fduwjj/197/base -> origin/gh/fduwjj/197/base 2025-10-10T00:37:13.0766685Z * [new branch] gh/fduwjj/197/head -> origin/gh/fduwjj/197/head 2025-10-10T00:37:13.0766966Z * [new branch] gh/fduwjj/197/orig -> origin/gh/fduwjj/197/orig 2025-10-10T00:37:13.0767244Z * [new branch] gh/fduwjj/198/base -> origin/gh/fduwjj/198/base 2025-10-10T00:37:13.0767511Z * [new branch] gh/fduwjj/198/head -> origin/gh/fduwjj/198/head 2025-10-10T00:37:13.0767651Z * [new branch] gh/fduwjj/198/orig -> origin/gh/fduwjj/198/orig 2025-10-10T00:37:13.0767790Z * [new branch] gh/fduwjj/199/base -> origin/gh/fduwjj/199/base 2025-10-10T00:37:13.0771927Z * [new branch] gh/fduwjj/199/head -> origin/gh/fduwjj/199/head 2025-10-10T00:37:13.0772501Z * [new branch] gh/fduwjj/199/orig -> origin/gh/fduwjj/199/orig 2025-10-10T00:37:13.0772674Z * [new branch] gh/fduwjj/200/base -> origin/gh/fduwjj/200/base 2025-10-10T00:37:13.0772822Z * [new branch] gh/fduwjj/200/head -> origin/gh/fduwjj/200/head 2025-10-10T00:37:13.0772955Z * [new branch] gh/fduwjj/200/orig -> origin/gh/fduwjj/200/orig 2025-10-10T00:37:13.0773126Z * [new branch] gh/fduwjj/201/base -> origin/gh/fduwjj/201/base 2025-10-10T00:37:13.0773268Z * [new branch] gh/fduwjj/201/head -> origin/gh/fduwjj/201/head 2025-10-10T00:37:13.0773402Z * [new branch] gh/fduwjj/201/orig -> origin/gh/fduwjj/201/orig 2025-10-10T00:37:13.0773587Z * [new branch] gh/fduwjj/202/base -> origin/gh/fduwjj/202/base 2025-10-10T00:37:13.0773999Z * [new branch] gh/fduwjj/202/head -> origin/gh/fduwjj/202/head 2025-10-10T00:37:13.0775197Z * [new branch] gh/fduwjj/202/orig -> origin/gh/fduwjj/202/orig 2025-10-10T00:37:13.0776306Z * [new branch] gh/fduwjj/203/base -> origin/gh/fduwjj/203/base 2025-10-10T00:37:13.0776870Z * [new branch] gh/fduwjj/203/head -> origin/gh/fduwjj/203/head 2025-10-10T00:37:13.0777777Z * [new branch] gh/fduwjj/203/orig -> origin/gh/fduwjj/203/orig 2025-10-10T00:37:13.0778924Z * [new branch] gh/fduwjj/204/base -> origin/gh/fduwjj/204/base 2025-10-10T00:37:13.0779195Z * [new branch] gh/fduwjj/204/head -> origin/gh/fduwjj/204/head 2025-10-10T00:37:13.0780261Z * [new branch] gh/fduwjj/204/orig -> origin/gh/fduwjj/204/orig 2025-10-10T00:37:13.0782926Z * [new branch] gh/fduwjj/205/base -> origin/gh/fduwjj/205/base 2025-10-10T00:37:13.0783570Z * [new branch] gh/fduwjj/205/head -> origin/gh/fduwjj/205/head 2025-10-10T00:37:13.0783734Z * [new branch] gh/fduwjj/205/orig -> origin/gh/fduwjj/205/orig 2025-10-10T00:37:13.0784031Z * [new branch] gh/fduwjj/206/base -> origin/gh/fduwjj/206/base 2025-10-10T00:37:13.0784181Z * [new branch] gh/fduwjj/206/head -> origin/gh/fduwjj/206/head 2025-10-10T00:37:13.0785219Z * [new branch] gh/fduwjj/206/orig -> origin/gh/fduwjj/206/orig 2025-10-10T00:37:13.0786542Z * [new branch] gh/fduwjj/207/base -> origin/gh/fduwjj/207/base 2025-10-10T00:37:13.0787164Z * [new branch] gh/fduwjj/207/head -> origin/gh/fduwjj/207/head 2025-10-10T00:37:13.0788423Z * [new branch] gh/fduwjj/207/orig -> origin/gh/fduwjj/207/orig 2025-10-10T00:37:13.0789310Z * [new branch] gh/fduwjj/208/base -> origin/gh/fduwjj/208/base 2025-10-10T00:37:13.0790309Z * [new branch] gh/fduwjj/208/head -> origin/gh/fduwjj/208/head 2025-10-10T00:37:13.0790556Z * [new branch] gh/fduwjj/208/orig -> origin/gh/fduwjj/208/orig 2025-10-10T00:37:13.0791769Z * [new branch] gh/fduwjj/209/base -> origin/gh/fduwjj/209/base 2025-10-10T00:37:13.0792069Z * [new branch] gh/fduwjj/209/head -> origin/gh/fduwjj/209/head 2025-10-10T00:37:13.0796101Z * [new branch] gh/fduwjj/209/orig -> origin/gh/fduwjj/209/orig 2025-10-10T00:37:13.0796287Z * [new branch] gh/fduwjj/210/base -> origin/gh/fduwjj/210/base 2025-10-10T00:37:13.0796430Z * [new branch] gh/fduwjj/210/head -> origin/gh/fduwjj/210/head 2025-10-10T00:37:13.0796565Z * [new branch] gh/fduwjj/210/orig -> origin/gh/fduwjj/210/orig 2025-10-10T00:37:13.0796719Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-10-10T00:37:13.0796890Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-10-10T00:37:13.0797735Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-10-10T00:37:13.0799082Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-10-10T00:37:13.0799290Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-10-10T00:37:13.0799693Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-10-10T00:37:13.0804740Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-10-10T00:37:13.0804911Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-10-10T00:37:13.0805044Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-10-10T00:37:13.0805199Z * [new branch] gh/fduwjj/214/base -> origin/gh/fduwjj/214/base 2025-10-10T00:37:13.0805329Z * [new branch] gh/fduwjj/214/head -> origin/gh/fduwjj/214/head 2025-10-10T00:37:13.0805463Z * [new branch] gh/fduwjj/214/orig -> origin/gh/fduwjj/214/orig 2025-10-10T00:37:13.0805912Z * [new branch] gh/fduwjj/215/base -> origin/gh/fduwjj/215/base 2025-10-10T00:37:13.0806088Z * [new branch] gh/fduwjj/215/head -> origin/gh/fduwjj/215/head 2025-10-10T00:37:13.0807051Z * [new branch] gh/fduwjj/215/orig -> origin/gh/fduwjj/215/orig 2025-10-10T00:37:13.0811545Z * [new branch] gh/fduwjj/216/base -> origin/gh/fduwjj/216/base 2025-10-10T00:37:13.0811721Z * [new branch] gh/fduwjj/216/head -> origin/gh/fduwjj/216/head 2025-10-10T00:37:13.0811859Z * [new branch] gh/fduwjj/216/orig -> origin/gh/fduwjj/216/orig 2025-10-10T00:37:13.0812164Z * [new branch] gh/fduwjj/217/base -> origin/gh/fduwjj/217/base 2025-10-10T00:37:13.0814270Z * [new branch] gh/fduwjj/217/head -> origin/gh/fduwjj/217/head 2025-10-10T00:37:13.0814586Z * [new branch] gh/fduwjj/217/orig -> origin/gh/fduwjj/217/orig 2025-10-10T00:37:13.0814755Z * [new branch] gh/fduwjj/218/base -> origin/gh/fduwjj/218/base 2025-10-10T00:37:13.0814901Z * [new branch] gh/fduwjj/218/head -> origin/gh/fduwjj/218/head 2025-10-10T00:37:13.0815216Z * [new branch] gh/fduwjj/218/orig -> origin/gh/fduwjj/218/orig 2025-10-10T00:37:13.0815511Z * [new branch] gh/fduwjj/219/base -> origin/gh/fduwjj/219/base 2025-10-10T00:37:13.0815688Z * [new branch] gh/fduwjj/219/head -> origin/gh/fduwjj/219/head 2025-10-10T00:37:13.0815834Z * [new branch] gh/fduwjj/219/orig -> origin/gh/fduwjj/219/orig 2025-10-10T00:37:13.0817149Z * [new branch] gh/fduwjj/220/base -> origin/gh/fduwjj/220/base 2025-10-10T00:37:13.0817328Z * [new branch] gh/fduwjj/220/head -> origin/gh/fduwjj/220/head 2025-10-10T00:37:13.0818006Z * [new branch] gh/fduwjj/220/orig -> origin/gh/fduwjj/220/orig 2025-10-10T00:37:13.0822088Z * [new branch] gh/fduwjj/221/base -> origin/gh/fduwjj/221/base 2025-10-10T00:37:13.0822278Z * [new branch] gh/fduwjj/221/head -> origin/gh/fduwjj/221/head 2025-10-10T00:37:13.0822594Z * [new branch] gh/fduwjj/221/orig -> origin/gh/fduwjj/221/orig 2025-10-10T00:37:13.0822745Z * [new branch] gh/fduwjj/222/base -> origin/gh/fduwjj/222/base 2025-10-10T00:37:13.0822883Z * [new branch] gh/fduwjj/222/head -> origin/gh/fduwjj/222/head 2025-10-10T00:37:13.0823022Z * [new branch] gh/fduwjj/222/orig -> origin/gh/fduwjj/222/orig 2025-10-10T00:37:13.0823654Z * [new branch] gh/fduwjj/223/base -> origin/gh/fduwjj/223/base 2025-10-10T00:37:13.0824251Z * [new branch] gh/fduwjj/223/head -> origin/gh/fduwjj/223/head 2025-10-10T00:37:13.0825118Z * [new branch] gh/fduwjj/223/orig -> origin/gh/fduwjj/223/orig 2025-10-10T00:37:13.0829886Z * [new branch] gh/fegin/313/base -> origin/gh/fegin/313/base 2025-10-10T00:37:13.0835491Z * [new branch] gh/fegin/313/head -> origin/gh/fegin/313/head 2025-10-10T00:37:13.0838957Z * [new branch] gh/fegin/313/orig -> origin/gh/fegin/313/orig 2025-10-10T00:37:13.0841360Z * [new branch] gh/fegin/314/base -> origin/gh/fegin/314/base 2025-10-10T00:37:13.0841631Z * [new branch] gh/fegin/314/head -> origin/gh/fegin/314/head 2025-10-10T00:37:13.0846409Z * [new branch] gh/fegin/314/orig -> origin/gh/fegin/314/orig 2025-10-10T00:37:13.0849407Z * [new branch] gh/fegin/315/base -> origin/gh/fegin/315/base 2025-10-10T00:37:13.0849685Z * [new branch] gh/fegin/315/head -> origin/gh/fegin/315/head 2025-10-10T00:37:13.0854493Z * [new branch] gh/fegin/315/orig -> origin/gh/fegin/315/orig 2025-10-10T00:37:13.0856530Z * [new branch] gh/fegin/316/base -> origin/gh/fegin/316/base 2025-10-10T00:37:13.0856785Z * [new branch] gh/fegin/316/head -> origin/gh/fegin/316/head 2025-10-10T00:37:13.0860364Z * [new branch] gh/fegin/316/orig -> origin/gh/fegin/316/orig 2025-10-10T00:37:13.0860681Z * [new branch] gh/fegin/317/base -> origin/gh/fegin/317/base 2025-10-10T00:37:13.0860839Z * [new branch] gh/fegin/317/head -> origin/gh/fegin/317/head 2025-10-10T00:37:13.0860964Z * [new branch] gh/fegin/317/orig -> origin/gh/fegin/317/orig 2025-10-10T00:37:13.0861281Z * [new branch] gh/fegin/318/base -> origin/gh/fegin/318/base 2025-10-10T00:37:13.0861414Z * [new branch] gh/fegin/318/head -> origin/gh/fegin/318/head 2025-10-10T00:37:13.0861675Z * [new branch] gh/fegin/318/orig -> origin/gh/fegin/318/orig 2025-10-10T00:37:13.0861815Z * [new branch] gh/fegin/319/base -> origin/gh/fegin/319/base 2025-10-10T00:37:13.0861968Z * [new branch] gh/fegin/319/head -> origin/gh/fegin/319/head 2025-10-10T00:37:13.0862101Z * [new branch] gh/fegin/319/orig -> origin/gh/fegin/319/orig 2025-10-10T00:37:13.0862235Z * [new branch] gh/fegin/320/base -> origin/gh/fegin/320/base 2025-10-10T00:37:13.0862361Z * [new branch] gh/fegin/320/head -> origin/gh/fegin/320/head 2025-10-10T00:37:13.0862488Z * [new branch] gh/fegin/320/orig -> origin/gh/fegin/320/orig 2025-10-10T00:37:13.0862625Z * [new branch] gh/fegin/321/base -> origin/gh/fegin/321/base 2025-10-10T00:37:13.0862752Z * [new branch] gh/fegin/321/head -> origin/gh/fegin/321/head 2025-10-10T00:37:13.0862890Z * [new branch] gh/fegin/321/orig -> origin/gh/fegin/321/orig 2025-10-10T00:37:13.0863016Z * [new branch] gh/fegin/322/base -> origin/gh/fegin/322/base 2025-10-10T00:37:13.0863139Z * [new branch] gh/fegin/322/head -> origin/gh/fegin/322/head 2025-10-10T00:37:13.0863328Z * [new branch] gh/fegin/322/orig -> origin/gh/fegin/322/orig 2025-10-10T00:37:13.0863451Z * [new branch] gh/fegin/323/base -> origin/gh/fegin/323/base 2025-10-10T00:37:13.0863579Z * [new branch] gh/fegin/323/head -> origin/gh/fegin/323/head 2025-10-10T00:37:13.0863704Z * [new branch] gh/fegin/324/base -> origin/gh/fegin/324/base 2025-10-10T00:37:13.0863841Z * [new branch] gh/fegin/324/head -> origin/gh/fegin/324/head 2025-10-10T00:37:13.0863969Z * [new branch] gh/fegin/324/orig -> origin/gh/fegin/324/orig 2025-10-10T00:37:13.0864131Z * [new branch] gh/fegin/325/base -> origin/gh/fegin/325/base 2025-10-10T00:37:13.0864289Z * [new branch] gh/fegin/325/head -> origin/gh/fegin/325/head 2025-10-10T00:37:13.0864435Z * [new branch] gh/fegin/325/orig -> origin/gh/fegin/325/orig 2025-10-10T00:37:13.0864584Z * [new branch] gh/fegin/326/base -> origin/gh/fegin/326/base 2025-10-10T00:37:13.0864927Z * [new branch] gh/fegin/326/head -> origin/gh/fegin/326/head 2025-10-10T00:37:13.0867015Z * [new branch] gh/fegin/326/orig -> origin/gh/fegin/326/orig 2025-10-10T00:37:13.0867184Z * [new branch] gh/fegin/327/base -> origin/gh/fegin/327/base 2025-10-10T00:37:13.0867534Z * [new branch] gh/fegin/327/head -> origin/gh/fegin/327/head 2025-10-10T00:37:13.0870727Z * [new branch] gh/fegin/327/orig -> origin/gh/fegin/327/orig 2025-10-10T00:37:13.0871089Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-10-10T00:37:13.0871342Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-10-10T00:37:13.0871528Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-10-10T00:37:13.0871831Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-10-10T00:37:13.0872249Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-10-10T00:37:13.0873183Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-10-10T00:37:13.0877603Z * [new branch] gh/fffrog/147/base -> origin/gh/fffrog/147/base 2025-10-10T00:37:13.0878119Z * [new branch] gh/fffrog/147/head -> origin/gh/fffrog/147/head 2025-10-10T00:37:13.0878337Z * [new branch] gh/fffrog/147/orig -> origin/gh/fffrog/147/orig 2025-10-10T00:37:13.0878490Z * [new branch] gh/fffrog/149/base -> origin/gh/fffrog/149/base 2025-10-10T00:37:13.0878715Z * [new branch] gh/fffrog/149/head -> origin/gh/fffrog/149/head 2025-10-10T00:37:13.0878868Z * [new branch] gh/fffrog/149/orig -> origin/gh/fffrog/149/orig 2025-10-10T00:37:13.0879446Z * [new branch] gh/fffrog/150/base -> origin/gh/fffrog/150/base 2025-10-10T00:37:13.0879623Z * [new branch] gh/fffrog/150/head -> origin/gh/fffrog/150/head 2025-10-10T00:37:13.0879757Z * [new branch] gh/fffrog/150/orig -> origin/gh/fffrog/150/orig 2025-10-10T00:37:13.0883783Z * [new branch] gh/fffrog/153/base -> origin/gh/fffrog/153/base 2025-10-10T00:37:13.0888741Z * [new branch] gh/fffrog/153/head -> origin/gh/fffrog/153/head 2025-10-10T00:37:13.0890825Z * [new branch] gh/fffrog/153/orig -> origin/gh/fffrog/153/orig 2025-10-10T00:37:13.0891097Z * [new branch] gh/fffrog/154/base -> origin/gh/fffrog/154/base 2025-10-10T00:37:13.0894699Z * [new branch] gh/fffrog/154/head -> origin/gh/fffrog/154/head 2025-10-10T00:37:13.0895115Z * [new branch] gh/fffrog/154/orig -> origin/gh/fffrog/154/orig 2025-10-10T00:37:13.0895384Z * [new branch] gh/fffrog/155/base -> origin/gh/fffrog/155/base 2025-10-10T00:37:13.0895549Z * [new branch] gh/fffrog/155/head -> origin/gh/fffrog/155/head 2025-10-10T00:37:13.0895679Z * [new branch] gh/fffrog/155/orig -> origin/gh/fffrog/155/orig 2025-10-10T00:37:13.0895831Z * [new branch] gh/fffrog/156/base -> origin/gh/fffrog/156/base 2025-10-10T00:37:13.0895965Z * [new branch] gh/fffrog/156/head -> origin/gh/fffrog/156/head 2025-10-10T00:37:13.0896097Z * [new branch] gh/fffrog/156/orig -> origin/gh/fffrog/156/orig 2025-10-10T00:37:13.0896226Z * [new branch] gh/fffrog/157/base -> origin/gh/fffrog/157/base 2025-10-10T00:37:13.0896363Z * [new branch] gh/fffrog/157/head -> origin/gh/fffrog/157/head 2025-10-10T00:37:13.0896498Z * [new branch] gh/fffrog/157/orig -> origin/gh/fffrog/157/orig 2025-10-10T00:37:13.0896627Z * [new branch] gh/fffrog/158/base -> origin/gh/fffrog/158/base 2025-10-10T00:37:13.0896760Z * [new branch] gh/fffrog/158/head -> origin/gh/fffrog/158/head 2025-10-10T00:37:13.0896887Z * [new branch] gh/fffrog/158/orig -> origin/gh/fffrog/158/orig 2025-10-10T00:37:13.0897023Z * [new branch] gh/fffrog/159/base -> origin/gh/fffrog/159/base 2025-10-10T00:37:13.0897153Z * [new branch] gh/fffrog/159/head -> origin/gh/fffrog/159/head 2025-10-10T00:37:13.0897280Z * [new branch] gh/fffrog/159/orig -> origin/gh/fffrog/159/orig 2025-10-10T00:37:13.0897420Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-10-10T00:37:13.0897548Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-10-10T00:37:13.0897727Z * [new branch] gh/fffrog/161/base -> origin/gh/fffrog/161/base 2025-10-10T00:37:13.0902207Z * [new branch] gh/fffrog/161/head -> origin/gh/fffrog/161/head 2025-10-10T00:37:13.0902385Z * [new branch] gh/fffrog/161/orig -> origin/gh/fffrog/161/orig 2025-10-10T00:37:13.0902533Z * [new branch] gh/fffrog/162/base -> origin/gh/fffrog/162/base 2025-10-10T00:37:13.0902843Z * [new branch] gh/fffrog/162/head -> origin/gh/fffrog/162/head 2025-10-10T00:37:13.0902995Z * [new branch] gh/fffrog/162/orig -> origin/gh/fffrog/162/orig 2025-10-10T00:37:13.0903145Z * [new branch] gh/fffrog/163/base -> origin/gh/fffrog/163/base 2025-10-10T00:37:13.0903300Z * [new branch] gh/fffrog/163/head -> origin/gh/fffrog/163/head 2025-10-10T00:37:13.0903452Z * [new branch] gh/fffrog/163/orig -> origin/gh/fffrog/163/orig 2025-10-10T00:37:13.0904857Z * [new branch] gh/fffrog/164/base -> origin/gh/fffrog/164/base 2025-10-10T00:37:13.0905486Z * [new branch] gh/fffrog/164/head -> origin/gh/fffrog/164/head 2025-10-10T00:37:13.0906111Z * [new branch] gh/fffrog/164/orig -> origin/gh/fffrog/164/orig 2025-10-10T00:37:13.0907517Z * [new branch] gh/fffrog/165/base -> origin/gh/fffrog/165/base 2025-10-10T00:37:13.0911698Z * [new branch] gh/fffrog/165/head -> origin/gh/fffrog/165/head 2025-10-10T00:37:13.0911937Z * [new branch] gh/fffrog/165/orig -> origin/gh/fffrog/165/orig 2025-10-10T00:37:13.0912091Z * [new branch] gh/fffrog/166/base -> origin/gh/fffrog/166/base 2025-10-10T00:37:13.0912237Z * [new branch] gh/fffrog/166/head -> origin/gh/fffrog/166/head 2025-10-10T00:37:13.0912369Z * [new branch] gh/fffrog/166/orig -> origin/gh/fffrog/166/orig 2025-10-10T00:37:13.0912548Z * [new branch] gh/fffrog/167/base -> origin/gh/fffrog/167/base 2025-10-10T00:37:13.0912690Z * [new branch] gh/fffrog/167/head -> origin/gh/fffrog/167/head 2025-10-10T00:37:13.0916054Z * [new branch] gh/fffrog/167/orig -> origin/gh/fffrog/167/orig 2025-10-10T00:37:13.0916395Z * [new branch] gh/fffrog/168/base -> origin/gh/fffrog/168/base 2025-10-10T00:37:13.0916566Z * [new branch] gh/fffrog/168/head -> origin/gh/fffrog/168/head 2025-10-10T00:37:13.0916714Z * [new branch] gh/fffrog/168/orig -> origin/gh/fffrog/168/orig 2025-10-10T00:37:13.0916859Z * [new branch] gh/fffrog/169/base -> origin/gh/fffrog/169/base 2025-10-10T00:37:13.0917016Z * [new branch] gh/fffrog/169/head -> origin/gh/fffrog/169/head 2025-10-10T00:37:13.0917164Z * [new branch] gh/fffrog/169/orig -> origin/gh/fffrog/169/orig 2025-10-10T00:37:13.0921721Z * [new branch] gh/fffrog/170/base -> origin/gh/fffrog/170/base 2025-10-10T00:37:13.0921976Z * [new branch] gh/fffrog/170/head -> origin/gh/fffrog/170/head 2025-10-10T00:37:13.0922164Z * [new branch] gh/fffrog/170/orig -> origin/gh/fffrog/170/orig 2025-10-10T00:37:13.0922291Z * [new branch] gh/fffrog/171/base -> origin/gh/fffrog/171/base 2025-10-10T00:37:13.0922557Z * [new branch] gh/fffrog/171/head -> origin/gh/fffrog/171/head 2025-10-10T00:37:13.0922700Z * [new branch] gh/fffrog/171/orig -> origin/gh/fffrog/171/orig 2025-10-10T00:37:13.0922894Z * [new branch] gh/fffrog/172/base -> origin/gh/fffrog/172/base 2025-10-10T00:37:13.0923105Z * [new branch] gh/fffrog/172/head -> origin/gh/fffrog/172/head 2025-10-10T00:37:13.0925669Z * [new branch] gh/fffrog/172/orig -> origin/gh/fffrog/172/orig 2025-10-10T00:37:13.0926048Z * [new branch] gh/fffrog/173/base -> origin/gh/fffrog/173/base 2025-10-10T00:37:13.0926185Z * [new branch] gh/fffrog/173/head -> origin/gh/fffrog/173/head 2025-10-10T00:37:13.0926414Z * [new branch] gh/fffrog/173/orig -> origin/gh/fffrog/173/orig 2025-10-10T00:37:13.0926549Z * [new branch] gh/fffrog/174/base -> origin/gh/fffrog/174/base 2025-10-10T00:37:13.0926951Z * [new branch] gh/fffrog/174/head -> origin/gh/fffrog/174/head 2025-10-10T00:37:13.0932940Z * [new branch] gh/fffrog/174/orig -> origin/gh/fffrog/174/orig 2025-10-10T00:37:13.0933129Z * [new branch] gh/fffrog/175/base -> origin/gh/fffrog/175/base 2025-10-10T00:37:13.0933282Z * [new branch] gh/fffrog/175/head -> origin/gh/fffrog/175/head 2025-10-10T00:37:13.0933431Z * [new branch] gh/fffrog/175/orig -> origin/gh/fffrog/175/orig 2025-10-10T00:37:13.0933613Z * [new branch] gh/fffrog/176/base -> origin/gh/fffrog/176/base 2025-10-10T00:37:13.0933757Z * [new branch] gh/fffrog/176/head -> origin/gh/fffrog/176/head 2025-10-10T00:37:13.0938321Z * [new branch] gh/fffrog/176/orig -> origin/gh/fffrog/176/orig 2025-10-10T00:37:13.0944121Z * [new branch] gh/fxdawnn/1/base -> origin/gh/fxdawnn/1/base 2025-10-10T00:37:13.0944772Z * [new branch] gh/fxdawnn/1/head -> origin/gh/fxdawnn/1/head 2025-10-10T00:37:13.0944948Z * [new branch] gh/fxdawnn/1/orig -> origin/gh/fxdawnn/1/orig 2025-10-10T00:37:13.0945104Z * [new branch] gh/fxdawnn/2/base -> origin/gh/fxdawnn/2/base 2025-10-10T00:37:13.0945256Z * [new branch] gh/fxdawnn/2/head -> origin/gh/fxdawnn/2/head 2025-10-10T00:37:13.0945403Z * [new branch] gh/fxdawnn/2/orig -> origin/gh/fxdawnn/2/orig 2025-10-10T00:37:13.0945723Z * [new branch] gh/fxdawnn/3/base -> origin/gh/fxdawnn/3/base 2025-10-10T00:37:13.0945887Z * [new branch] gh/fxdawnn/3/head -> origin/gh/fxdawnn/3/head 2025-10-10T00:37:13.0946023Z * [new branch] gh/fxdawnn/3/orig -> origin/gh/fxdawnn/3/orig 2025-10-10T00:37:13.0946158Z * [new branch] gh/fxdawnn/4/base -> origin/gh/fxdawnn/4/base 2025-10-10T00:37:13.0946579Z * [new branch] gh/fxdawnn/4/orig -> origin/gh/fxdawnn/4/orig 2025-10-10T00:37:13.0946754Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-10-10T00:37:13.0946909Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-10-10T00:37:13.0947053Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-10-10T00:37:13.0947202Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-10-10T00:37:13.0947350Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-10-10T00:37:13.0947493Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-10-10T00:37:13.0947646Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-10-10T00:37:13.0947918Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-10-10T00:37:13.0950868Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-10-10T00:37:13.0956430Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-10-10T00:37:13.0958978Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-10-10T00:37:13.0959158Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-10-10T00:37:13.0959349Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-10-10T00:37:13.0959494Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-10-10T00:37:13.0959648Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-10-10T00:37:13.0959809Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-10-10T00:37:13.0960170Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-10-10T00:37:13.0960327Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-10-10T00:37:13.0960482Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-10-10T00:37:13.0960641Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-10-10T00:37:13.0960792Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-10-10T00:37:13.0960938Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-10-10T00:37:13.0961088Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-10-10T00:37:13.0964923Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-10-10T00:37:13.0967034Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-10-10T00:37:13.0967336Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-10-10T00:37:13.0973305Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-10-10T00:37:13.0973488Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-10-10T00:37:13.0973907Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-10-10T00:37:13.0974081Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-10-10T00:37:13.0974377Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-10-10T00:37:13.0974535Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-10-10T00:37:13.0974695Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-10-10T00:37:13.0974846Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-10-10T00:37:13.0975002Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-10-10T00:37:13.0975155Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-10-10T00:37:13.0975310Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-10-10T00:37:13.0975457Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-10-10T00:37:13.0975606Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-10-10T00:37:13.0975767Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-10-10T00:37:13.0975915Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-10-10T00:37:13.0981428Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-10-10T00:37:13.0981764Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-10-10T00:37:13.0982028Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-10-10T00:37:13.0982260Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-10-10T00:37:13.0982442Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-10-10T00:37:13.0982596Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-10-10T00:37:13.0982875Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-10-10T00:37:13.0983107Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-10-10T00:37:13.0983337Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-10-10T00:37:13.0983526Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-10-10T00:37:13.0983943Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-10-10T00:37:13.0984127Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-10-10T00:37:13.0984275Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-10-10T00:37:13.0985681Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-10-10T00:37:13.0985867Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-10-10T00:37:13.0986061Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-10-10T00:37:13.0986447Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-10-10T00:37:13.0986798Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-10-10T00:37:13.0986993Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-10-10T00:37:13.0994655Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-10-10T00:37:13.0994971Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-10-10T00:37:13.0995394Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-10-10T00:37:13.0995575Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-10-10T00:37:13.0995765Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-10-10T00:37:13.0996087Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-10-10T00:37:13.0996246Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-10-10T00:37:13.0996534Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-10-10T00:37:13.0996684Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-10-10T00:37:13.0996843Z * [new branch] gh/guangyey/194/base -> origin/gh/guangyey/194/base 2025-10-10T00:37:13.0996984Z * [new branch] gh/guangyey/194/head -> origin/gh/guangyey/194/head 2025-10-10T00:37:13.0997130Z * [new branch] gh/guangyey/194/orig -> origin/gh/guangyey/194/orig 2025-10-10T00:37:13.0997276Z * [new branch] gh/guangyey/195/base -> origin/gh/guangyey/195/base 2025-10-10T00:37:13.1002687Z * [new branch] gh/guangyey/195/head -> origin/gh/guangyey/195/head 2025-10-10T00:37:13.1002889Z * [new branch] gh/guangyey/195/orig -> origin/gh/guangyey/195/orig 2025-10-10T00:37:13.1003041Z * [new branch] gh/guangyey/201/base -> origin/gh/guangyey/201/base 2025-10-10T00:37:13.1003195Z * [new branch] gh/guangyey/201/head -> origin/gh/guangyey/201/head 2025-10-10T00:37:13.1003339Z * [new branch] gh/guangyey/201/orig -> origin/gh/guangyey/201/orig 2025-10-10T00:37:13.1009172Z * [new branch] gh/guangyey/202/base -> origin/gh/guangyey/202/base 2025-10-10T00:37:13.1009525Z * [new branch] gh/guangyey/202/head -> origin/gh/guangyey/202/head 2025-10-10T00:37:13.1009762Z * [new branch] gh/guangyey/202/orig -> origin/gh/guangyey/202/orig 2025-10-10T00:37:13.1009954Z * [new branch] gh/guangyey/203/base -> origin/gh/guangyey/203/base 2025-10-10T00:37:13.1010185Z * [new branch] gh/guangyey/203/head -> origin/gh/guangyey/203/head 2025-10-10T00:37:13.1010377Z * [new branch] gh/guangyey/203/orig -> origin/gh/guangyey/203/orig 2025-10-10T00:37:13.1010541Z * [new branch] gh/guangyey/205/base -> origin/gh/guangyey/205/base 2025-10-10T00:37:13.1010690Z * [new branch] gh/guangyey/205/head -> origin/gh/guangyey/205/head 2025-10-10T00:37:13.1011067Z * [new branch] gh/guangyey/205/orig -> origin/gh/guangyey/205/orig 2025-10-10T00:37:13.1011652Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-10-10T00:37:13.1011836Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-10-10T00:37:13.1011987Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-10-10T00:37:13.1012127Z * [new branch] gh/guangyey/209/base -> origin/gh/guangyey/209/base 2025-10-10T00:37:13.1012285Z * [new branch] gh/guangyey/209/head -> origin/gh/guangyey/209/head 2025-10-10T00:37:13.1012433Z * [new branch] gh/guangyey/209/orig -> origin/gh/guangyey/209/orig 2025-10-10T00:37:13.1018568Z * [new branch] gh/guangyey/210/base -> origin/gh/guangyey/210/base 2025-10-10T00:37:13.1018923Z * [new branch] gh/guangyey/210/head -> origin/gh/guangyey/210/head 2025-10-10T00:37:13.1019177Z * [new branch] gh/guangyey/210/orig -> origin/gh/guangyey/210/orig 2025-10-10T00:37:13.1019811Z * [new branch] gh/guangyey/211/base -> origin/gh/guangyey/211/base 2025-10-10T00:37:13.1020012Z * [new branch] gh/guangyey/211/head -> origin/gh/guangyey/211/head 2025-10-10T00:37:13.1020169Z * [new branch] gh/guangyey/211/orig -> origin/gh/guangyey/211/orig 2025-10-10T00:37:13.1020365Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-10-10T00:37:13.1020669Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-10-10T00:37:13.1020824Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-10-10T00:37:13.1021024Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-10-10T00:37:13.1021211Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-10-10T00:37:13.1021400Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-10-10T00:37:13.1021579Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-10-10T00:37:13.1021762Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-10-10T00:37:13.1022564Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-10-10T00:37:13.1023719Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-10-10T00:37:13.1024339Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-10-10T00:37:13.1025325Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-10-10T00:37:13.1026395Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-10-10T00:37:13.1026970Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-10-10T00:37:13.1032001Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-10-10T00:37:13.1032229Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-10-10T00:37:13.1032414Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-10-10T00:37:13.1032589Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-10-10T00:37:13.1032836Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-10-10T00:37:13.1033016Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-10-10T00:37:13.1033197Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-10-10T00:37:13.1033992Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-10-10T00:37:13.1039180Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-10-10T00:37:13.1042109Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-10-10T00:37:13.1045188Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-10-10T00:37:13.1045483Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-10-10T00:37:13.1053057Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-10-10T00:37:13.1055383Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-10-10T00:37:13.1060490Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-10-10T00:37:13.1060818Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-10-10T00:37:13.1061086Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-10-10T00:37:13.1061413Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-10-10T00:37:13.1061585Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-10-10T00:37:13.1061749Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-10-10T00:37:13.1062094Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-10-10T00:37:13.1062279Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-10-10T00:37:13.1062436Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-10-10T00:37:13.1062597Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-10-10T00:37:13.1062753Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-10-10T00:37:13.1062913Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-10-10T00:37:13.1063069Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-10-10T00:37:13.1063223Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-10-10T00:37:13.1063389Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-10-10T00:37:13.1063542Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-10-10T00:37:13.1063702Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-10-10T00:37:13.1063854Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-10-10T00:37:13.1064009Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-10-10T00:37:13.1064169Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-10-10T00:37:13.1064321Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-10-10T00:37:13.1064483Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-10-10T00:37:13.1064635Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-10-10T00:37:13.1064797Z * [new branch] gh/guilhermeleobas/237/base -> origin/gh/guilhermeleobas/237/base 2025-10-10T00:37:13.1064949Z * [new branch] gh/guilhermeleobas/237/head -> origin/gh/guilhermeleobas/237/head 2025-10-10T00:37:13.1065102Z * [new branch] gh/guilhermeleobas/237/orig -> origin/gh/guilhermeleobas/237/orig 2025-10-10T00:37:13.1065304Z * [new branch] gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base 2025-10-10T00:37:13.1065454Z * [new branch] gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head 2025-10-10T00:37:13.1065612Z * [new branch] gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig 2025-10-10T00:37:13.1065763Z * [new branch] gh/guilhermeleobas/246/base -> origin/gh/guilhermeleobas/246/base 2025-10-10T00:37:13.1065922Z * [new branch] gh/guilhermeleobas/246/head -> origin/gh/guilhermeleobas/246/head 2025-10-10T00:37:13.1066086Z * [new branch] gh/guilhermeleobas/246/orig -> origin/gh/guilhermeleobas/246/orig 2025-10-10T00:37:13.1066237Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-10-10T00:37:13.1066602Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-10-10T00:37:13.1066767Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-10-10T00:37:13.1066936Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-10-10T00:37:13.1067107Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-10-10T00:37:13.1067636Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-10-10T00:37:13.1068583Z * [new branch] gh/guilhermeleobas/249/base -> origin/gh/guilhermeleobas/249/base 2025-10-10T00:37:13.1069128Z * [new branch] gh/guilhermeleobas/249/head -> origin/gh/guilhermeleobas/249/head 2025-10-10T00:37:13.1073422Z * [new branch] gh/guilhermeleobas/249/orig -> origin/gh/guilhermeleobas/249/orig 2025-10-10T00:37:13.1078204Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-10-10T00:37:13.1082433Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-10-10T00:37:13.1087097Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-10-10T00:37:13.1088529Z * [new branch] gh/henrylhtsang/150/base -> origin/gh/henrylhtsang/150/base 2025-10-10T00:37:13.1088722Z * [new branch] gh/henrylhtsang/150/head -> origin/gh/henrylhtsang/150/head 2025-10-10T00:37:13.1088891Z * [new branch] gh/henrylhtsang/150/orig -> origin/gh/henrylhtsang/150/orig 2025-10-10T00:37:13.1089086Z * [new branch] gh/henrylhtsang/151/base -> origin/gh/henrylhtsang/151/base 2025-10-10T00:37:13.1089282Z * [new branch] gh/henrylhtsang/151/head -> origin/gh/henrylhtsang/151/head 2025-10-10T00:37:13.1089439Z * [new branch] gh/henrylhtsang/151/orig -> origin/gh/henrylhtsang/151/orig 2025-10-10T00:37:13.1089601Z * [new branch] gh/henrylhtsang/152/base -> origin/gh/henrylhtsang/152/base 2025-10-10T00:37:13.1089758Z * [new branch] gh/henrylhtsang/152/head -> origin/gh/henrylhtsang/152/head 2025-10-10T00:37:13.1089911Z * [new branch] gh/henrylhtsang/152/orig -> origin/gh/henrylhtsang/152/orig 2025-10-10T00:37:13.1090051Z * [new branch] gh/henrylhtsang/153/base -> origin/gh/henrylhtsang/153/base 2025-10-10T00:37:13.1090198Z * [new branch] gh/henrylhtsang/153/head -> origin/gh/henrylhtsang/153/head 2025-10-10T00:37:13.1090338Z * [new branch] gh/henrylhtsang/153/orig -> origin/gh/henrylhtsang/153/orig 2025-10-10T00:37:13.1090495Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-10-10T00:37:13.1090629Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-10-10T00:37:13.1090758Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-10-10T00:37:13.1090883Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-10-10T00:37:13.1091176Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-10-10T00:37:13.1091308Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-10-10T00:37:13.1091444Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-10-10T00:37:13.1091575Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-10-10T00:37:13.1091716Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-10-10T00:37:13.1091852Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-10-10T00:37:13.1091989Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-10-10T00:37:13.1092148Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-10-10T00:37:13.1092313Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-10-10T00:37:13.1097748Z * [new branch] gh/isuruf/147/base -> origin/gh/isuruf/147/base 2025-10-10T00:37:13.1097930Z * [new branch] gh/isuruf/147/head -> origin/gh/isuruf/147/head 2025-10-10T00:37:13.1098076Z * [new branch] gh/isuruf/147/orig -> origin/gh/isuruf/147/orig 2025-10-10T00:37:13.1098220Z * [new branch] gh/isuruf/148/base -> origin/gh/isuruf/148/base 2025-10-10T00:37:13.1098372Z * [new branch] gh/isuruf/148/head -> origin/gh/isuruf/148/head 2025-10-10T00:37:13.1098690Z * [new branch] gh/isuruf/148/orig -> origin/gh/isuruf/148/orig 2025-10-10T00:37:13.1098852Z * [new branch] gh/isuruf/149/base -> origin/gh/isuruf/149/base 2025-10-10T00:37:13.1098994Z * [new branch] gh/isuruf/149/head -> origin/gh/isuruf/149/head 2025-10-10T00:37:13.1099138Z * [new branch] gh/isuruf/149/orig -> origin/gh/isuruf/149/orig 2025-10-10T00:37:13.1100300Z * [new branch] gh/isuruf/150/base -> origin/gh/isuruf/150/base 2025-10-10T00:37:13.1100893Z * [new branch] gh/isuruf/150/head -> origin/gh/isuruf/150/head 2025-10-10T00:37:13.1101330Z * [new branch] gh/isuruf/150/orig -> origin/gh/isuruf/150/orig 2025-10-10T00:37:13.1102398Z * [new branch] gh/isuruf/151/base -> origin/gh/isuruf/151/base 2025-10-10T00:37:13.1102762Z * [new branch] gh/isuruf/151/head -> origin/gh/isuruf/151/head 2025-10-10T00:37:13.1103979Z * [new branch] gh/isuruf/151/orig -> origin/gh/isuruf/151/orig 2025-10-10T00:37:13.1104258Z * [new branch] gh/isuruf/152/base -> origin/gh/isuruf/152/base 2025-10-10T00:37:13.1105376Z * [new branch] gh/isuruf/152/head -> origin/gh/isuruf/152/head 2025-10-10T00:37:13.1105606Z * [new branch] gh/isuruf/152/orig -> origin/gh/isuruf/152/orig 2025-10-10T00:37:13.1109613Z * [new branch] gh/isuruf/153/base -> origin/gh/isuruf/153/base 2025-10-10T00:37:13.1114024Z * [new branch] gh/isuruf/153/head -> origin/gh/isuruf/153/head 2025-10-10T00:37:13.1118932Z * [new branch] gh/isuruf/153/orig -> origin/gh/isuruf/153/orig 2025-10-10T00:37:13.1123919Z * [new branch] gh/isuruf/154/base -> origin/gh/isuruf/154/base 2025-10-10T00:37:13.1126079Z * [new branch] gh/isuruf/154/head -> origin/gh/isuruf/154/head 2025-10-10T00:37:13.1126363Z * [new branch] gh/isuruf/154/orig -> origin/gh/isuruf/154/orig 2025-10-10T00:37:13.1133021Z * [new branch] gh/isuruf/155/base -> origin/gh/isuruf/155/base 2025-10-10T00:37:13.1138640Z * [new branch] gh/isuruf/155/head -> origin/gh/isuruf/155/head 2025-10-10T00:37:13.1140700Z * [new branch] gh/isuruf/155/orig -> origin/gh/isuruf/155/orig 2025-10-10T00:37:13.1141055Z * [new branch] gh/isuruf/156/base -> origin/gh/isuruf/156/base 2025-10-10T00:37:13.1141207Z * [new branch] gh/isuruf/156/head -> origin/gh/isuruf/156/head 2025-10-10T00:37:13.1141353Z * [new branch] gh/isuruf/156/orig -> origin/gh/isuruf/156/orig 2025-10-10T00:37:13.1141492Z * [new branch] gh/isuruf/157/base -> origin/gh/isuruf/157/base 2025-10-10T00:37:13.1141625Z * [new branch] gh/isuruf/157/head -> origin/gh/isuruf/157/head 2025-10-10T00:37:13.1141771Z * [new branch] gh/isuruf/157/orig -> origin/gh/isuruf/157/orig 2025-10-10T00:37:13.1141909Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-10-10T00:37:13.1142042Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-10-10T00:37:13.1142178Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-10-10T00:37:13.1142330Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-10-10T00:37:13.1142472Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-10-10T00:37:13.1142607Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-10-10T00:37:13.1142739Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-10-10T00:37:13.1142878Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-10-10T00:37:13.1143076Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-10-10T00:37:13.1143219Z * [new branch] gh/jamesjwu/186/base -> origin/gh/jamesjwu/186/base 2025-10-10T00:37:13.1143354Z * [new branch] gh/jamesjwu/186/head -> origin/gh/jamesjwu/186/head 2025-10-10T00:37:13.1143493Z * [new branch] gh/jamesjwu/186/orig -> origin/gh/jamesjwu/186/orig 2025-10-10T00:37:13.1143630Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-10-10T00:37:13.1143779Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-10-10T00:37:13.1143935Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-10-10T00:37:13.1144085Z * [new branch] gh/jamesjwu/189/base -> origin/gh/jamesjwu/189/base 2025-10-10T00:37:13.1144237Z * [new branch] gh/jamesjwu/189/head -> origin/gh/jamesjwu/189/head 2025-10-10T00:37:13.1144378Z * [new branch] gh/jamesjwu/189/orig -> origin/gh/jamesjwu/189/orig 2025-10-10T00:37:13.1144525Z * [new branch] gh/jamesjwu/190/base -> origin/gh/jamesjwu/190/base 2025-10-10T00:37:13.1144681Z * [new branch] gh/jamesjwu/190/head -> origin/gh/jamesjwu/190/head 2025-10-10T00:37:13.1144826Z * [new branch] gh/jamesjwu/190/orig -> origin/gh/jamesjwu/190/orig 2025-10-10T00:37:13.1144978Z * [new branch] gh/jamesjwu/191/base -> origin/gh/jamesjwu/191/base 2025-10-10T00:37:13.1145112Z * [new branch] gh/jamesjwu/191/head -> origin/gh/jamesjwu/191/head 2025-10-10T00:37:13.1145252Z * [new branch] gh/jamesjwu/191/orig -> origin/gh/jamesjwu/191/orig 2025-10-10T00:37:13.1145389Z * [new branch] gh/jamesjwu/192/base -> origin/gh/jamesjwu/192/base 2025-10-10T00:37:13.1145524Z * [new branch] gh/jamesjwu/192/head -> origin/gh/jamesjwu/192/head 2025-10-10T00:37:13.1145666Z * [new branch] gh/jamesjwu/193/base -> origin/gh/jamesjwu/193/base 2025-10-10T00:37:13.1145802Z * [new branch] gh/jamesjwu/193/head -> origin/gh/jamesjwu/193/head 2025-10-10T00:37:13.1145948Z * [new branch] gh/jamesjwu/193/orig -> origin/gh/jamesjwu/193/orig 2025-10-10T00:37:13.1146126Z * [new branch] gh/jamesjwu/194/base -> origin/gh/jamesjwu/194/base 2025-10-10T00:37:13.1146463Z * [new branch] gh/jamesjwu/194/head -> origin/gh/jamesjwu/194/head 2025-10-10T00:37:13.1146628Z * [new branch] gh/jamesjwu/194/orig -> origin/gh/jamesjwu/194/orig 2025-10-10T00:37:13.1146782Z * [new branch] gh/jamesjwu/195/base -> origin/gh/jamesjwu/195/base 2025-10-10T00:37:13.1146941Z * [new branch] gh/jamesjwu/195/head -> origin/gh/jamesjwu/195/head 2025-10-10T00:37:13.1147233Z * [new branch] gh/jamesjwu/195/orig -> origin/gh/jamesjwu/195/orig 2025-10-10T00:37:13.1147456Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-10-10T00:37:13.1148938Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-10-10T00:37:13.1149269Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-10-10T00:37:13.1149735Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-10-10T00:37:13.1151934Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-10-10T00:37:13.1152112Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-10-10T00:37:13.1152254Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-10-10T00:37:13.1158702Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-10-10T00:37:13.1164377Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-10-10T00:37:13.1168657Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-10-10T00:37:13.1170568Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-10-10T00:37:13.1170835Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-10-10T00:37:13.1175155Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-10-10T00:37:13.1175428Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-10-10T00:37:13.1181388Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-10-10T00:37:13.1181587Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-10-10T00:37:13.1181759Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-10-10T00:37:13.1181911Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-10-10T00:37:13.1182052Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-10-10T00:37:13.1182199Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-10-10T00:37:13.1182349Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-10-10T00:37:13.1182498Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-10-10T00:37:13.1182640Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-10-10T00:37:13.1182780Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-10-10T00:37:13.1182929Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-10-10T00:37:13.1183070Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-10-10T00:37:13.1183215Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-10-10T00:37:13.1183351Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-10-10T00:37:13.1183490Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-10-10T00:37:13.1183788Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-10-10T00:37:13.1183928Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-10-10T00:37:13.1184090Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-10-10T00:37:13.1184237Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-10-10T00:37:13.1184386Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-10-10T00:37:13.1184533Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-10-10T00:37:13.1184678Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-10-10T00:37:13.1184829Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-10-10T00:37:13.1184973Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-10-10T00:37:13.1185128Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-10-10T00:37:13.1185269Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-10-10T00:37:13.1185416Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-10-10T00:37:13.1185558Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-10-10T00:37:13.1185699Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-10-10T00:37:13.1185890Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-10-10T00:37:13.1186038Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-10-10T00:37:13.1186183Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-10-10T00:37:13.1186519Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-10-10T00:37:13.1186683Z * [new branch] gh/janeyx99/304/base -> origin/gh/janeyx99/304/base 2025-10-10T00:37:13.1186824Z * [new branch] gh/janeyx99/304/head -> origin/gh/janeyx99/304/head 2025-10-10T00:37:13.1186971Z * [new branch] gh/janeyx99/304/orig -> origin/gh/janeyx99/304/orig 2025-10-10T00:37:13.1187383Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-10-10T00:37:13.1188140Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-10-10T00:37:13.1189511Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-10-10T00:37:13.1189656Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-10-10T00:37:13.1193590Z * [new branch] gh/janeyx99/307/base -> origin/gh/janeyx99/307/base 2025-10-10T00:37:13.1193767Z * [new branch] gh/janeyx99/307/head -> origin/gh/janeyx99/307/head 2025-10-10T00:37:13.1193930Z * [new branch] gh/janeyx99/307/orig -> origin/gh/janeyx99/307/orig 2025-10-10T00:37:13.1194063Z * [new branch] gh/janeyx99/308/base -> origin/gh/janeyx99/308/base 2025-10-10T00:37:13.1194199Z * [new branch] gh/janeyx99/308/head -> origin/gh/janeyx99/308/head 2025-10-10T00:37:13.1194374Z * [new branch] gh/janeyx99/308/orig -> origin/gh/janeyx99/308/orig 2025-10-10T00:37:13.1195936Z * [new branch] gh/janeyx99/309/base -> origin/gh/janeyx99/309/base 2025-10-10T00:37:13.1196279Z * [new branch] gh/janeyx99/309/head -> origin/gh/janeyx99/309/head 2025-10-10T00:37:13.1196821Z * [new branch] gh/janeyx99/309/orig -> origin/gh/janeyx99/309/orig 2025-10-10T00:37:13.1198985Z * [new branch] gh/janeyx99/310/base -> origin/gh/janeyx99/310/base 2025-10-10T00:37:13.1199492Z * [new branch] gh/janeyx99/310/head -> origin/gh/janeyx99/310/head 2025-10-10T00:37:13.1199659Z * [new branch] gh/janeyx99/310/orig -> origin/gh/janeyx99/310/orig 2025-10-10T00:37:13.1199943Z * [new branch] gh/janeyx99/311/base -> origin/gh/janeyx99/311/base 2025-10-10T00:37:13.1201285Z * [new branch] gh/janeyx99/311/head -> origin/gh/janeyx99/311/head 2025-10-10T00:37:13.1201612Z * [new branch] gh/janeyx99/311/orig -> origin/gh/janeyx99/311/orig 2025-10-10T00:37:13.1202120Z * [new branch] gh/janeyx99/312/base -> origin/gh/janeyx99/312/base 2025-10-10T00:37:13.1204495Z * [new branch] gh/janeyx99/312/head -> origin/gh/janeyx99/312/head 2025-10-10T00:37:13.1204832Z * [new branch] gh/janeyx99/312/orig -> origin/gh/janeyx99/312/orig 2025-10-10T00:37:13.1205062Z * [new branch] gh/janeyx99/313/base -> origin/gh/janeyx99/313/base 2025-10-10T00:37:13.1205243Z * [new branch] gh/janeyx99/313/head -> origin/gh/janeyx99/313/head 2025-10-10T00:37:13.1205571Z * [new branch] gh/janeyx99/313/orig -> origin/gh/janeyx99/313/orig 2025-10-10T00:37:13.1207097Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-10-10T00:37:13.1207335Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-10-10T00:37:13.1209469Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-10-10T00:37:13.1209797Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-10-10T00:37:13.1210109Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-10-10T00:37:13.1210585Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-10-10T00:37:13.1211967Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-10-10T00:37:13.1212311Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-10-10T00:37:13.1215182Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-10-10T00:37:13.1215362Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-10-10T00:37:13.1215531Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-10-10T00:37:13.1216572Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-10-10T00:37:13.1216846Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-10-10T00:37:13.1216990Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-10-10T00:37:13.1218052Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-10-10T00:37:13.1218299Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-10-10T00:37:13.1220716Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-10-10T00:37:13.1220886Z * [new branch] gh/jansel/532/base -> origin/gh/jansel/532/base 2025-10-10T00:37:13.1221034Z * [new branch] gh/jansel/532/head -> origin/gh/jansel/532/head 2025-10-10T00:37:13.1221582Z * [new branch] gh/jansel/532/orig -> origin/gh/jansel/532/orig 2025-10-10T00:37:13.1225405Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-10-10T00:37:13.1225590Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-10-10T00:37:13.1225728Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-10-10T00:37:13.1225867Z * [new branch] gh/jansel/534/base -> origin/gh/jansel/534/base 2025-10-10T00:37:13.1226043Z * [new branch] gh/jansel/534/head -> origin/gh/jansel/534/head 2025-10-10T00:37:13.1226575Z * [new branch] gh/jansel/534/orig -> origin/gh/jansel/534/orig 2025-10-10T00:37:13.1232408Z * [new branch] gh/jansel/535/base -> origin/gh/jansel/535/base 2025-10-10T00:37:13.1232589Z * [new branch] gh/jansel/535/head -> origin/gh/jansel/535/head 2025-10-10T00:37:13.1232742Z * [new branch] gh/jansel/535/orig -> origin/gh/jansel/535/orig 2025-10-10T00:37:13.1232923Z * [new branch] gh/jansel/536/base -> origin/gh/jansel/536/base 2025-10-10T00:37:13.1233094Z * [new branch] gh/jansel/536/head -> origin/gh/jansel/536/head 2025-10-10T00:37:13.1233261Z * [new branch] gh/jansel/536/orig -> origin/gh/jansel/536/orig 2025-10-10T00:37:13.1233416Z * [new branch] gh/jansel/537/base -> origin/gh/jansel/537/base 2025-10-10T00:37:13.1233564Z * [new branch] gh/jansel/537/head -> origin/gh/jansel/537/head 2025-10-10T00:37:13.1237314Z * [new branch] gh/jansel/537/orig -> origin/gh/jansel/537/orig 2025-10-10T00:37:13.1237661Z * [new branch] gh/jansel/538/base -> origin/gh/jansel/538/base 2025-10-10T00:37:13.1237803Z * [new branch] gh/jansel/538/head -> origin/gh/jansel/538/head 2025-10-10T00:37:13.1237942Z * [new branch] gh/jansel/538/orig -> origin/gh/jansel/538/orig 2025-10-10T00:37:13.1238194Z * [new branch] gh/jansel/539/base -> origin/gh/jansel/539/base 2025-10-10T00:37:13.1238327Z * [new branch] gh/jansel/539/head -> origin/gh/jansel/539/head 2025-10-10T00:37:13.1238469Z * [new branch] gh/jansel/539/orig -> origin/gh/jansel/539/orig 2025-10-10T00:37:13.1240892Z * [new branch] gh/jansel/540/base -> origin/gh/jansel/540/base 2025-10-10T00:37:13.1241053Z * [new branch] gh/jansel/540/head -> origin/gh/jansel/540/head 2025-10-10T00:37:13.1241182Z * [new branch] gh/jansel/540/orig -> origin/gh/jansel/540/orig 2025-10-10T00:37:13.1241320Z * [new branch] gh/jansel/541/base -> origin/gh/jansel/541/base 2025-10-10T00:37:13.1244572Z * [new branch] gh/jansel/541/head -> origin/gh/jansel/541/head 2025-10-10T00:37:13.1245021Z * [new branch] gh/jansel/541/orig -> origin/gh/jansel/541/orig 2025-10-10T00:37:13.1245198Z * [new branch] gh/jansel/542/base -> origin/gh/jansel/542/base 2025-10-10T00:37:13.1245354Z * [new branch] gh/jansel/542/head -> origin/gh/jansel/542/head 2025-10-10T00:37:13.1245500Z * [new branch] gh/jansel/542/orig -> origin/gh/jansel/542/orig 2025-10-10T00:37:13.1248185Z * [new branch] gh/jansel/543/base -> origin/gh/jansel/543/base 2025-10-10T00:37:13.1248331Z * [new branch] gh/jansel/543/head -> origin/gh/jansel/543/head 2025-10-10T00:37:13.1248561Z * [new branch] gh/jansel/543/orig -> origin/gh/jansel/543/orig 2025-10-10T00:37:13.1254546Z * [new branch] gh/jansel/544/base -> origin/gh/jansel/544/base 2025-10-10T00:37:13.1259316Z * [new branch] gh/jansel/544/head -> origin/gh/jansel/544/head 2025-10-10T00:37:13.1263713Z * [new branch] gh/jansel/544/orig -> origin/gh/jansel/544/orig 2025-10-10T00:37:13.1263922Z * [new branch] gh/jansel/545/base -> origin/gh/jansel/545/base 2025-10-10T00:37:13.1264069Z * [new branch] gh/jansel/545/head -> origin/gh/jansel/545/head 2025-10-10T00:37:13.1264218Z * [new branch] gh/jansel/545/orig -> origin/gh/jansel/545/orig 2025-10-10T00:37:13.1264359Z * [new branch] gh/jansel/546/base -> origin/gh/jansel/546/base 2025-10-10T00:37:13.1264728Z * [new branch] gh/jansel/546/head -> origin/gh/jansel/546/head 2025-10-10T00:37:13.1264865Z * [new branch] gh/jansel/546/orig -> origin/gh/jansel/546/orig 2025-10-10T00:37:13.1265010Z * [new branch] gh/jansel/547/base -> origin/gh/jansel/547/base 2025-10-10T00:37:13.1265145Z * [new branch] gh/jansel/547/head -> origin/gh/jansel/547/head 2025-10-10T00:37:13.1265282Z * [new branch] gh/jansel/547/orig -> origin/gh/jansel/547/orig 2025-10-10T00:37:13.1265429Z * [new branch] gh/jansel/548/base -> origin/gh/jansel/548/base 2025-10-10T00:37:13.1265570Z * [new branch] gh/jansel/548/head -> origin/gh/jansel/548/head 2025-10-10T00:37:13.1265716Z * [new branch] gh/jansel/548/orig -> origin/gh/jansel/548/orig 2025-10-10T00:37:13.1265889Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-10-10T00:37:13.1266063Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-10-10T00:37:13.1266221Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-10-10T00:37:13.1266543Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-10-10T00:37:13.1266723Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-10-10T00:37:13.1266874Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-10-10T00:37:13.1267168Z * [new branch] gh/jbschlosser/251/base -> origin/gh/jbschlosser/251/base 2025-10-10T00:37:13.1267319Z * [new branch] gh/jbschlosser/251/head -> origin/gh/jbschlosser/251/head 2025-10-10T00:37:13.1267481Z * [new branch] gh/jbschlosser/251/orig -> origin/gh/jbschlosser/251/orig 2025-10-10T00:37:13.1273058Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-10-10T00:37:13.1275084Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-10-10T00:37:13.1275267Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-10-10T00:37:13.1275454Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-10-10T00:37:13.1275610Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-10-10T00:37:13.1275777Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-10-10T00:37:13.1275938Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-10-10T00:37:13.1276095Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-10-10T00:37:13.1276251Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-10-10T00:37:13.1276410Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-10-10T00:37:13.1276562Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-10-10T00:37:13.1276721Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-10-10T00:37:13.1282917Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-10-10T00:37:13.1283119Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-10-10T00:37:13.1283307Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-10-10T00:37:13.1283468Z * [new branch] gh/jiayisunx/71/base -> origin/gh/jiayisunx/71/base 2025-10-10T00:37:13.1283675Z * [new branch] gh/jiayisunx/71/head -> origin/gh/jiayisunx/71/head 2025-10-10T00:37:13.1283833Z * [new branch] gh/jiayisunx/71/orig -> origin/gh/jiayisunx/71/orig 2025-10-10T00:37:13.1284140Z * [new branch] gh/jiayisunx/72/base -> origin/gh/jiayisunx/72/base 2025-10-10T00:37:13.1284276Z * [new branch] gh/jiayisunx/72/head -> origin/gh/jiayisunx/72/head 2025-10-10T00:37:13.1284424Z * [new branch] gh/jiayisunx/72/orig -> origin/gh/jiayisunx/72/orig 2025-10-10T00:37:13.1284745Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-10-10T00:37:13.1289554Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-10-10T00:37:13.1289785Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-10-10T00:37:13.1289945Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-10-10T00:37:13.1290252Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-10-10T00:37:13.1290432Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-10-10T00:37:13.1290701Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-10-10T00:37:13.1290874Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-10-10T00:37:13.1291028Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-10-10T00:37:13.1291183Z * [new branch] gh/jiayisunx/80/base -> origin/gh/jiayisunx/80/base 2025-10-10T00:37:13.1291560Z * [new branch] gh/jiayisunx/80/head -> origin/gh/jiayisunx/80/head 2025-10-10T00:37:13.1296321Z * [new branch] gh/jiayisunx/80/orig -> origin/gh/jiayisunx/80/orig 2025-10-10T00:37:13.1300979Z * [new branch] gh/jiayisunx/81/base -> origin/gh/jiayisunx/81/base 2025-10-10T00:37:13.1301176Z * [new branch] gh/jiayisunx/81/head -> origin/gh/jiayisunx/81/head 2025-10-10T00:37:13.1301335Z * [new branch] gh/jiayisunx/81/orig -> origin/gh/jiayisunx/81/orig 2025-10-10T00:37:13.1301514Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-10-10T00:37:13.1301669Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-10-10T00:37:13.1301825Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-10-10T00:37:13.1301975Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-10-10T00:37:13.1302151Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-10-10T00:37:13.1302311Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-10-10T00:37:13.1302461Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-10-10T00:37:13.1302608Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-10-10T00:37:13.1302752Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-10-10T00:37:13.1305928Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-10-10T00:37:13.1310562Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-10-10T00:37:13.1316802Z * [new branch] gh/karthickai/3/base -> origin/gh/karthickai/3/base 2025-10-10T00:37:13.1322403Z * [new branch] gh/karthickai/3/head -> origin/gh/karthickai/3/head 2025-10-10T00:37:13.1324148Z * [new branch] gh/karthickai/3/orig -> origin/gh/karthickai/3/orig 2025-10-10T00:37:13.1324396Z * [new branch] gh/karthickai/4/base -> origin/gh/karthickai/4/base 2025-10-10T00:37:13.1324644Z * [new branch] gh/karthickai/4/head -> origin/gh/karthickai/4/head 2025-10-10T00:37:13.1324839Z * [new branch] gh/karthickai/4/orig -> origin/gh/karthickai/4/orig 2025-10-10T00:37:13.1325240Z * [new branch] gh/karthickai/5/base -> origin/gh/karthickai/5/base 2025-10-10T00:37:13.1325430Z * [new branch] gh/karthickai/5/head -> origin/gh/karthickai/5/head 2025-10-10T00:37:13.1325579Z * [new branch] gh/karthickai/5/orig -> origin/gh/karthickai/5/orig 2025-10-10T00:37:13.1325726Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-10-10T00:37:13.1325879Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-10-10T00:37:13.1326036Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-10-10T00:37:13.1326209Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-10-10T00:37:13.1326373Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-10-10T00:37:13.1326540Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-10-10T00:37:13.1326724Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-10-10T00:37:13.1326890Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-10-10T00:37:13.1327045Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-10-10T00:37:13.1327204Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-10-10T00:37:13.1327351Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-10-10T00:37:13.1327570Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-10-10T00:37:13.1327734Z * [new branch] gh/kurtamohler/51/base -> origin/gh/kurtamohler/51/base 2025-10-10T00:37:13.1327888Z * [new branch] gh/kurtamohler/51/head -> origin/gh/kurtamohler/51/head 2025-10-10T00:37:13.1328050Z * [new branch] gh/kurtamohler/51/orig -> origin/gh/kurtamohler/51/orig 2025-10-10T00:37:13.1328203Z * [new branch] gh/kurtamohler/52/base -> origin/gh/kurtamohler/52/base 2025-10-10T00:37:13.1328367Z * [new branch] gh/kurtamohler/52/head -> origin/gh/kurtamohler/52/head 2025-10-10T00:37:13.1328514Z * [new branch] gh/kurtamohler/52/orig -> origin/gh/kurtamohler/52/orig 2025-10-10T00:37:13.1328664Z * [new branch] gh/kurtamohler/53/base -> origin/gh/kurtamohler/53/base 2025-10-10T00:37:13.1332063Z * [new branch] gh/kurtamohler/53/head -> origin/gh/kurtamohler/53/head 2025-10-10T00:37:13.1332380Z * [new branch] gh/kurtamohler/53/orig -> origin/gh/kurtamohler/53/orig 2025-10-10T00:37:13.1332550Z * [new branch] gh/kurtamohler/54/base -> origin/gh/kurtamohler/54/base 2025-10-10T00:37:13.1332772Z * [new branch] gh/kurtamohler/54/head -> origin/gh/kurtamohler/54/head 2025-10-10T00:37:13.1333012Z * [new branch] gh/kurtamohler/54/orig -> origin/gh/kurtamohler/54/orig 2025-10-10T00:37:13.1333151Z * [new branch] gh/kurtamohler/55/base -> origin/gh/kurtamohler/55/base 2025-10-10T00:37:13.1333377Z * [new branch] gh/kurtamohler/55/head -> origin/gh/kurtamohler/55/head 2025-10-10T00:37:13.1333603Z * [new branch] gh/kurtamohler/55/orig -> origin/gh/kurtamohler/55/orig 2025-10-10T00:37:13.1333841Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-10-10T00:37:13.1335771Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-10-10T00:37:13.1340571Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-10-10T00:37:13.1340771Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-10-10T00:37:13.1340919Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-10-10T00:37:13.1341335Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-10-10T00:37:13.1341484Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-10-10T00:37:13.1341636Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-10-10T00:37:13.1341790Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-10-10T00:37:13.1342098Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-10-10T00:37:13.1342274Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-10-10T00:37:13.1343830Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-10-10T00:37:13.1344001Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-10-10T00:37:13.1344378Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-10-10T00:37:13.1345386Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-10-10T00:37:13.1346403Z * [new branch] gh/kwen2501/222/base -> origin/gh/kwen2501/222/base 2025-10-10T00:37:13.1347071Z * [new branch] gh/kwen2501/222/head -> origin/gh/kwen2501/222/head 2025-10-10T00:37:13.1347557Z * [new branch] gh/kwen2501/222/orig -> origin/gh/kwen2501/222/orig 2025-10-10T00:37:13.1349179Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-10-10T00:37:13.1349336Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-10-10T00:37:13.1350423Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-10-10T00:37:13.1350855Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-10-10T00:37:13.1351890Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-10-10T00:37:13.1352300Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-10-10T00:37:13.1353250Z * [new branch] gh/kwen2501/230/base -> origin/gh/kwen2501/230/base 2025-10-10T00:37:13.1353832Z * [new branch] gh/kwen2501/230/head -> origin/gh/kwen2501/230/head 2025-10-10T00:37:13.1354801Z * [new branch] gh/kwen2501/230/orig -> origin/gh/kwen2501/230/orig 2025-10-10T00:37:13.1355838Z * [new branch] gh/kwen2501/231/base -> origin/gh/kwen2501/231/base 2025-10-10T00:37:13.1356055Z * [new branch] gh/kwen2501/231/head -> origin/gh/kwen2501/231/head 2025-10-10T00:37:13.1357649Z * [new branch] gh/kwen2501/231/orig -> origin/gh/kwen2501/231/orig 2025-10-10T00:37:13.1357833Z * [new branch] gh/kwen2501/232/base -> origin/gh/kwen2501/232/base 2025-10-10T00:37:13.1358924Z * [new branch] gh/kwen2501/232/head -> origin/gh/kwen2501/232/head 2025-10-10T00:37:13.1359071Z * [new branch] gh/kwen2501/232/orig -> origin/gh/kwen2501/232/orig 2025-10-10T00:37:13.1360381Z * [new branch] gh/kwen2501/233/base -> origin/gh/kwen2501/233/base 2025-10-10T00:37:13.1360694Z * [new branch] gh/kwen2501/233/head -> origin/gh/kwen2501/233/head 2025-10-10T00:37:13.1361646Z * [new branch] gh/kwen2501/233/orig -> origin/gh/kwen2501/233/orig 2025-10-10T00:37:13.1363106Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-10-10T00:37:13.1363299Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-10-10T00:37:13.1364370Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-10-10T00:37:13.1365232Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-10-10T00:37:13.1365549Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-10-10T00:37:13.1366605Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-10-10T00:37:13.1367207Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-10-10T00:37:13.1368022Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-10-10T00:37:13.1368658Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-10-10T00:37:13.1369581Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-10-10T00:37:13.1370027Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-10-10T00:37:13.1370720Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-10-10T00:37:13.1372161Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-10-10T00:37:13.1372428Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-10-10T00:37:13.1373394Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-10-10T00:37:13.1374809Z * [new branch] gh/kwen2501/239/base -> origin/gh/kwen2501/239/base 2025-10-10T00:37:13.1375100Z * [new branch] gh/kwen2501/239/head -> origin/gh/kwen2501/239/head 2025-10-10T00:37:13.1375703Z * [new branch] gh/kwen2501/239/orig -> origin/gh/kwen2501/239/orig 2025-10-10T00:37:13.1376771Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-10-10T00:37:13.1377053Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-10-10T00:37:13.1378033Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-10-10T00:37:13.1378733Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-10-10T00:37:13.1379280Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-10-10T00:37:13.1380409Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-10-10T00:37:13.1380997Z * [new branch] gh/kwen2501/242/base -> origin/gh/kwen2501/242/base 2025-10-10T00:37:13.1381900Z * [new branch] gh/kwen2501/242/head -> origin/gh/kwen2501/242/head 2025-10-10T00:37:13.1382199Z * [new branch] gh/kwen2501/242/orig -> origin/gh/kwen2501/242/orig 2025-10-10T00:37:13.1383377Z * [new branch] gh/kwen2501/243/base -> origin/gh/kwen2501/243/base 2025-10-10T00:37:13.1384109Z * [new branch] gh/kwen2501/243/head -> origin/gh/kwen2501/243/head 2025-10-10T00:37:13.1384548Z * [new branch] gh/kwen2501/243/orig -> origin/gh/kwen2501/243/orig 2025-10-10T00:37:13.1385631Z * [new branch] gh/kwen2501/244/base -> origin/gh/kwen2501/244/base 2025-10-10T00:37:13.1386211Z * [new branch] gh/kwen2501/244/head -> origin/gh/kwen2501/244/head 2025-10-10T00:37:13.1386905Z * [new branch] gh/kwen2501/244/orig -> origin/gh/kwen2501/244/orig 2025-10-10T00:37:13.1388170Z * [new branch] gh/kwen2501/245/base -> origin/gh/kwen2501/245/base 2025-10-10T00:37:13.1388411Z * [new branch] gh/kwen2501/245/head -> origin/gh/kwen2501/245/head 2025-10-10T00:37:13.1390580Z * [new branch] gh/kwen2501/245/orig -> origin/gh/kwen2501/245/orig 2025-10-10T00:37:13.1390913Z * [new branch] gh/kwen2501/246/base -> origin/gh/kwen2501/246/base 2025-10-10T00:37:13.1391146Z * [new branch] gh/kwen2501/246/head -> origin/gh/kwen2501/246/head 2025-10-10T00:37:13.1391372Z * [new branch] gh/kwen2501/246/orig -> origin/gh/kwen2501/246/orig 2025-10-10T00:37:13.1392100Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-10-10T00:37:13.1392853Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-10-10T00:37:13.1393315Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-10-10T00:37:13.1395222Z * [new branch] gh/kwen2501/248/base -> origin/gh/kwen2501/248/base 2025-10-10T00:37:13.1395536Z * [new branch] gh/kwen2501/248/head -> origin/gh/kwen2501/248/head 2025-10-10T00:37:13.1395964Z * [new branch] gh/kwen2501/248/orig -> origin/gh/kwen2501/248/orig 2025-10-10T00:37:13.1398994Z * [new branch] gh/kwen2501/249/base -> origin/gh/kwen2501/249/base 2025-10-10T00:37:13.1399165Z * [new branch] gh/kwen2501/249/head -> origin/gh/kwen2501/249/head 2025-10-10T00:37:13.1399327Z * [new branch] gh/kwen2501/249/orig -> origin/gh/kwen2501/249/orig 2025-10-10T00:37:13.1399472Z * [new branch] gh/kwen2501/250/base -> origin/gh/kwen2501/250/base 2025-10-10T00:37:13.1399889Z * [new branch] gh/kwen2501/250/head -> origin/gh/kwen2501/250/head 2025-10-10T00:37:13.1400454Z * [new branch] gh/kwen2501/250/orig -> origin/gh/kwen2501/250/orig 2025-10-10T00:37:13.1405033Z * [new branch] gh/kwen2501/251/base -> origin/gh/kwen2501/251/base 2025-10-10T00:37:13.1405349Z * [new branch] gh/kwen2501/251/head -> origin/gh/kwen2501/251/head 2025-10-10T00:37:13.1405503Z * [new branch] gh/kwen2501/251/orig -> origin/gh/kwen2501/251/orig 2025-10-10T00:37:13.1405649Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-10-10T00:37:13.1405789Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-10-10T00:37:13.1405947Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-10-10T00:37:13.1406321Z * [new branch] gh/kwen2501/253/base -> origin/gh/kwen2501/253/base 2025-10-10T00:37:13.1406554Z * [new branch] gh/kwen2501/253/head -> origin/gh/kwen2501/253/head 2025-10-10T00:37:13.1407586Z * [new branch] gh/kwen2501/253/orig -> origin/gh/kwen2501/253/orig 2025-10-10T00:37:13.1411503Z * [new branch] gh/kwen2501/254/base -> origin/gh/kwen2501/254/base 2025-10-10T00:37:13.1411701Z * [new branch] gh/kwen2501/254/head -> origin/gh/kwen2501/254/head 2025-10-10T00:37:13.1411845Z * [new branch] gh/kwen2501/254/orig -> origin/gh/kwen2501/254/orig 2025-10-10T00:37:13.1412000Z * [new branch] gh/kwen2501/255/base -> origin/gh/kwen2501/255/base 2025-10-10T00:37:13.1412132Z * [new branch] gh/kwen2501/255/head -> origin/gh/kwen2501/255/head 2025-10-10T00:37:13.1412352Z * [new branch] gh/kwen2501/255/orig -> origin/gh/kwen2501/255/orig 2025-10-10T00:37:13.1413328Z * [new branch] gh/kwen2501/256/base -> origin/gh/kwen2501/256/base 2025-10-10T00:37:13.1413906Z * [new branch] gh/kwen2501/256/head -> origin/gh/kwen2501/256/head 2025-10-10T00:37:13.1418895Z * [new branch] gh/kwen2501/256/orig -> origin/gh/kwen2501/256/orig 2025-10-10T00:37:13.1419083Z * [new branch] gh/kwen2501/257/base -> origin/gh/kwen2501/257/base 2025-10-10T00:37:13.1419265Z * [new branch] gh/kwen2501/257/head -> origin/gh/kwen2501/257/head 2025-10-10T00:37:13.1419404Z * [new branch] gh/kwen2501/257/orig -> origin/gh/kwen2501/257/orig 2025-10-10T00:37:13.1419548Z * [new branch] gh/kwen2501/258/base -> origin/gh/kwen2501/258/base 2025-10-10T00:37:13.1419687Z * [new branch] gh/kwen2501/258/head -> origin/gh/kwen2501/258/head 2025-10-10T00:37:13.1419986Z * [new branch] gh/kwen2501/258/orig -> origin/gh/kwen2501/258/orig 2025-10-10T00:37:13.1420175Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-10-10T00:37:13.1421041Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-10-10T00:37:13.1421501Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-10-10T00:37:13.1422760Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-10-10T00:37:13.1424075Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-10-10T00:37:13.1424357Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-10-10T00:37:13.1424710Z * [new branch] gh/kwen2501/261/base -> origin/gh/kwen2501/261/base 2025-10-10T00:37:13.1425781Z * [new branch] gh/kwen2501/261/head -> origin/gh/kwen2501/261/head 2025-10-10T00:37:13.1426101Z * [new branch] gh/kwen2501/261/orig -> origin/gh/kwen2501/261/orig 2025-10-10T00:37:13.1427659Z * [new branch] gh/kwen2501/262/base -> origin/gh/kwen2501/262/base 2025-10-10T00:37:13.1428003Z * [new branch] gh/kwen2501/262/head -> origin/gh/kwen2501/262/head 2025-10-10T00:37:13.1429087Z * [new branch] gh/kwen2501/262/orig -> origin/gh/kwen2501/262/orig 2025-10-10T00:37:13.1433806Z * [new branch] gh/kwen2501/263/base -> origin/gh/kwen2501/263/base 2025-10-10T00:37:13.1433959Z * [new branch] gh/kwen2501/263/head -> origin/gh/kwen2501/263/head 2025-10-10T00:37:13.1434105Z * [new branch] gh/kwen2501/263/orig -> origin/gh/kwen2501/263/orig 2025-10-10T00:37:13.1434242Z * [new branch] gh/kwen2501/264/base -> origin/gh/kwen2501/264/base 2025-10-10T00:37:13.1434392Z * [new branch] gh/kwen2501/264/head -> origin/gh/kwen2501/264/head 2025-10-10T00:37:13.1434528Z * [new branch] gh/kwen2501/264/orig -> origin/gh/kwen2501/264/orig 2025-10-10T00:37:13.1439394Z * [new branch] gh/kwen2501/265/base -> origin/gh/kwen2501/265/base 2025-10-10T00:37:13.1445142Z * [new branch] gh/kwen2501/265/head -> origin/gh/kwen2501/265/head 2025-10-10T00:37:13.1445361Z * [new branch] gh/kwen2501/265/orig -> origin/gh/kwen2501/265/orig 2025-10-10T00:37:13.1445581Z * [new branch] gh/kwen2501/266/base -> origin/gh/kwen2501/266/base 2025-10-10T00:37:13.1445741Z * [new branch] gh/kwen2501/266/head -> origin/gh/kwen2501/266/head 2025-10-10T00:37:13.1445879Z * [new branch] gh/kwen2501/266/orig -> origin/gh/kwen2501/266/orig 2025-10-10T00:37:13.1446093Z * [new branch] gh/kwen2501/267/base -> origin/gh/kwen2501/267/base 2025-10-10T00:37:13.1446248Z * [new branch] gh/kwen2501/267/head -> origin/gh/kwen2501/267/head 2025-10-10T00:37:13.1449755Z * [new branch] gh/kwen2501/267/orig -> origin/gh/kwen2501/267/orig 2025-10-10T00:37:13.1450018Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-10-10T00:37:13.1450178Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-10-10T00:37:13.1450401Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-10-10T00:37:13.1450663Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-10-10T00:37:13.1450858Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-10-10T00:37:13.1451029Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-10-10T00:37:13.1451173Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-10-10T00:37:13.1451547Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-10-10T00:37:13.1455197Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-10-10T00:37:13.1455430Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-10-10T00:37:13.1455599Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-10-10T00:37:13.1455824Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-10-10T00:37:13.1455989Z * [new branch] gh/kwen2501/272/base -> origin/gh/kwen2501/272/base 2025-10-10T00:37:13.1456135Z * [new branch] gh/kwen2501/272/head -> origin/gh/kwen2501/272/head 2025-10-10T00:37:13.1456358Z * [new branch] gh/kwen2501/272/orig -> origin/gh/kwen2501/272/orig 2025-10-10T00:37:13.1456510Z * [new branch] gh/kwen2501/273/base -> origin/gh/kwen2501/273/base 2025-10-10T00:37:13.1456658Z * [new branch] gh/kwen2501/273/head -> origin/gh/kwen2501/273/head 2025-10-10T00:37:13.1456799Z * [new branch] gh/kwen2501/273/orig -> origin/gh/kwen2501/273/orig 2025-10-10T00:37:13.1457286Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-10-10T00:37:13.1457464Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-10-10T00:37:13.1457798Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-10-10T00:37:13.1458181Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-10-10T00:37:13.1458706Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-10-10T00:37:13.1459544Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-10-10T00:37:13.1460827Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-10-10T00:37:13.1461238Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-10-10T00:37:13.1462269Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-10-10T00:37:13.1463062Z * [new branch] gh/laithsakka/263/base -> origin/gh/laithsakka/263/base 2025-10-10T00:37:13.1463549Z * [new branch] gh/laithsakka/263/head -> origin/gh/laithsakka/263/head 2025-10-10T00:37:13.1464553Z * [new branch] gh/laithsakka/263/orig -> origin/gh/laithsakka/263/orig 2025-10-10T00:37:13.1465323Z * [new branch] gh/laithsakka/264/base -> origin/gh/laithsakka/264/base 2025-10-10T00:37:13.1465844Z * [new branch] gh/laithsakka/264/head -> origin/gh/laithsakka/264/head 2025-10-10T00:37:13.1466943Z * [new branch] gh/laithsakka/264/orig -> origin/gh/laithsakka/264/orig 2025-10-10T00:37:13.1467550Z * [new branch] gh/laithsakka/268/base -> origin/gh/laithsakka/268/base 2025-10-10T00:37:13.1469077Z * [new branch] gh/laithsakka/268/head -> origin/gh/laithsakka/268/head 2025-10-10T00:37:13.1469250Z * [new branch] gh/laithsakka/268/orig -> origin/gh/laithsakka/268/orig 2025-10-10T00:37:13.1469759Z * [new branch] gh/laithsakka/269/base -> origin/gh/laithsakka/269/base 2025-10-10T00:37:13.1470763Z * [new branch] gh/laithsakka/269/head -> origin/gh/laithsakka/269/head 2025-10-10T00:37:13.1471318Z * [new branch] gh/laithsakka/269/orig -> origin/gh/laithsakka/269/orig 2025-10-10T00:37:13.1473490Z * [new branch] gh/laithsakka/271/base -> origin/gh/laithsakka/271/base 2025-10-10T00:37:13.1474138Z * [new branch] gh/laithsakka/271/head -> origin/gh/laithsakka/271/head 2025-10-10T00:37:13.1474499Z * [new branch] gh/laithsakka/271/orig -> origin/gh/laithsakka/271/orig 2025-10-10T00:37:13.1475021Z * [new branch] gh/laithsakka/272/base -> origin/gh/laithsakka/272/base 2025-10-10T00:37:13.1475627Z * [new branch] gh/laithsakka/272/head -> origin/gh/laithsakka/272/head 2025-10-10T00:37:13.1476585Z * [new branch] gh/laithsakka/272/orig -> origin/gh/laithsakka/272/orig 2025-10-10T00:37:13.1478179Z * [new branch] gh/laithsakka/273/base -> origin/gh/laithsakka/273/base 2025-10-10T00:37:13.1478524Z * [new branch] gh/laithsakka/273/head -> origin/gh/laithsakka/273/head 2025-10-10T00:37:13.1479659Z * [new branch] gh/laithsakka/273/orig -> origin/gh/laithsakka/273/orig 2025-10-10T00:37:13.1480406Z * [new branch] gh/laithsakka/274/base -> origin/gh/laithsakka/274/base 2025-10-10T00:37:13.1483634Z * [new branch] gh/laithsakka/274/head -> origin/gh/laithsakka/274/head 2025-10-10T00:37:13.1483844Z * [new branch] gh/laithsakka/274/orig -> origin/gh/laithsakka/274/orig 2025-10-10T00:37:13.1484014Z * [new branch] gh/laithsakka/275/base -> origin/gh/laithsakka/275/base 2025-10-10T00:37:13.1484203Z * [new branch] gh/laithsakka/275/head -> origin/gh/laithsakka/275/head 2025-10-10T00:37:13.1484581Z * [new branch] gh/laithsakka/275/orig -> origin/gh/laithsakka/275/orig 2025-10-10T00:37:13.1485131Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-10-10T00:37:13.1488813Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-10-10T00:37:13.1489012Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-10-10T00:37:13.1489176Z * [new branch] gh/laithsakka/277/base -> origin/gh/laithsakka/277/base 2025-10-10T00:37:13.1489329Z * [new branch] gh/laithsakka/277/head -> origin/gh/laithsakka/277/head 2025-10-10T00:37:13.1489503Z * [new branch] gh/laithsakka/277/orig -> origin/gh/laithsakka/277/orig 2025-10-10T00:37:13.1489847Z * [new branch] gh/laithsakka/278/base -> origin/gh/laithsakka/278/base 2025-10-10T00:37:13.1490888Z * [new branch] gh/laithsakka/278/head -> origin/gh/laithsakka/278/head 2025-10-10T00:37:13.1491836Z * [new branch] gh/laithsakka/278/orig -> origin/gh/laithsakka/278/orig 2025-10-10T00:37:13.1492252Z * [new branch] gh/laithsakka/279/base -> origin/gh/laithsakka/279/base 2025-10-10T00:37:13.1495387Z * [new branch] gh/laithsakka/279/head -> origin/gh/laithsakka/279/head 2025-10-10T00:37:13.1495754Z * [new branch] gh/laithsakka/279/orig -> origin/gh/laithsakka/279/orig 2025-10-10T00:37:13.1495966Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-10-10T00:37:13.1496181Z * [new branch] gh/laithsakka/280/base -> origin/gh/laithsakka/280/base 2025-10-10T00:37:13.1500366Z * [new branch] gh/laithsakka/280/head -> origin/gh/laithsakka/280/head 2025-10-10T00:37:13.1500705Z * [new branch] gh/laithsakka/280/orig -> origin/gh/laithsakka/280/orig 2025-10-10T00:37:13.1500946Z * [new branch] gh/laithsakka/281/base -> origin/gh/laithsakka/281/base 2025-10-10T00:37:13.1501189Z * [new branch] gh/laithsakka/281/head -> origin/gh/laithsakka/281/head 2025-10-10T00:37:13.1501383Z * [new branch] gh/laithsakka/281/orig -> origin/gh/laithsakka/281/orig 2025-10-10T00:37:13.1502138Z * [new branch] gh/laithsakka/282/base -> origin/gh/laithsakka/282/base 2025-10-10T00:37:13.1503618Z * [new branch] gh/laithsakka/282/head -> origin/gh/laithsakka/282/head 2025-10-10T00:37:13.1503845Z * [new branch] gh/laithsakka/282/orig -> origin/gh/laithsakka/282/orig 2025-10-10T00:37:13.1505662Z * [new branch] gh/laithsakka/283/base -> origin/gh/laithsakka/283/base 2025-10-10T00:37:13.1506019Z * [new branch] gh/laithsakka/283/head -> origin/gh/laithsakka/283/head 2025-10-10T00:37:13.1509973Z * [new branch] gh/laithsakka/283/orig -> origin/gh/laithsakka/283/orig 2025-10-10T00:37:13.1510164Z * [new branch] gh/laithsakka/284/base -> origin/gh/laithsakka/284/base 2025-10-10T00:37:13.1510822Z * [new branch] gh/laithsakka/284/head -> origin/gh/laithsakka/284/head 2025-10-10T00:37:13.1511137Z * [new branch] gh/laithsakka/284/orig -> origin/gh/laithsakka/284/orig 2025-10-10T00:37:13.1511422Z * [new branch] gh/laithsakka/285/base -> origin/gh/laithsakka/285/base 2025-10-10T00:37:13.1511589Z * [new branch] gh/laithsakka/285/head -> origin/gh/laithsakka/285/head 2025-10-10T00:37:13.1511855Z * [new branch] gh/laithsakka/285/orig -> origin/gh/laithsakka/285/orig 2025-10-10T00:37:13.1516269Z * [new branch] gh/laithsakka/286/base -> origin/gh/laithsakka/286/base 2025-10-10T00:37:13.1516525Z * [new branch] gh/laithsakka/286/head -> origin/gh/laithsakka/286/head 2025-10-10T00:37:13.1516707Z * [new branch] gh/laithsakka/286/orig -> origin/gh/laithsakka/286/orig 2025-10-10T00:37:13.1516947Z * [new branch] gh/laithsakka/287/base -> origin/gh/laithsakka/287/base 2025-10-10T00:37:13.1517530Z * [new branch] gh/laithsakka/287/head -> origin/gh/laithsakka/287/head 2025-10-10T00:37:13.1518105Z * [new branch] gh/laithsakka/287/orig -> origin/gh/laithsakka/287/orig 2025-10-10T00:37:13.1523043Z * [new branch] gh/laithsakka/288/base -> origin/gh/laithsakka/288/base 2025-10-10T00:37:13.1526468Z * [new branch] gh/laithsakka/288/head -> origin/gh/laithsakka/288/head 2025-10-10T00:37:13.1526770Z * [new branch] gh/laithsakka/288/orig -> origin/gh/laithsakka/288/orig 2025-10-10T00:37:13.1532992Z * [new branch] gh/laithsakka/289/base -> origin/gh/laithsakka/289/base 2025-10-10T00:37:13.1537365Z * [new branch] gh/laithsakka/289/head -> origin/gh/laithsakka/289/head 2025-10-10T00:37:13.1540063Z * [new branch] gh/laithsakka/289/orig -> origin/gh/laithsakka/289/orig 2025-10-10T00:37:13.1540364Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-10-10T00:37:13.1543963Z * [new branch] gh/laithsakka/290/base -> origin/gh/laithsakka/290/base 2025-10-10T00:37:13.1550027Z * [new branch] gh/laithsakka/290/head -> origin/gh/laithsakka/290/head 2025-10-10T00:37:13.1554649Z * [new branch] gh/laithsakka/290/orig -> origin/gh/laithsakka/290/orig 2025-10-10T00:37:13.1560350Z * [new branch] gh/laithsakka/291/base -> origin/gh/laithsakka/291/base 2025-10-10T00:37:13.1562208Z * [new branch] gh/laithsakka/291/head -> origin/gh/laithsakka/291/head 2025-10-10T00:37:13.1562476Z * [new branch] gh/laithsakka/291/orig -> origin/gh/laithsakka/291/orig 2025-10-10T00:37:13.1567512Z * [new branch] gh/laithsakka/292/base -> origin/gh/laithsakka/292/base 2025-10-10T00:37:13.1569502Z * [new branch] gh/laithsakka/292/head -> origin/gh/laithsakka/292/head 2025-10-10T00:37:13.1569812Z * [new branch] gh/laithsakka/292/orig -> origin/gh/laithsakka/292/orig 2025-10-10T00:37:13.1569967Z * [new branch] gh/laithsakka/293/base -> origin/gh/laithsakka/293/base 2025-10-10T00:37:13.1570173Z * [new branch] gh/laithsakka/293/head -> origin/gh/laithsakka/293/head 2025-10-10T00:37:13.1574019Z * [new branch] gh/laithsakka/293/orig -> origin/gh/laithsakka/293/orig 2025-10-10T00:37:13.1574502Z * [new branch] gh/laithsakka/294/base -> origin/gh/laithsakka/294/base 2025-10-10T00:37:13.1579252Z * [new branch] gh/laithsakka/294/head -> origin/gh/laithsakka/294/head 2025-10-10T00:37:13.1581255Z * [new branch] gh/laithsakka/294/orig -> origin/gh/laithsakka/294/orig 2025-10-10T00:37:13.1581428Z * [new branch] gh/laithsakka/295/base -> origin/gh/laithsakka/295/base 2025-10-10T00:37:13.1581598Z * [new branch] gh/laithsakka/295/head -> origin/gh/laithsakka/295/head 2025-10-10T00:37:13.1581761Z * [new branch] gh/laithsakka/295/orig -> origin/gh/laithsakka/295/orig 2025-10-10T00:37:13.1581904Z * [new branch] gh/laithsakka/296/base -> origin/gh/laithsakka/296/base 2025-10-10T00:37:13.1582047Z * [new branch] gh/laithsakka/296/head -> origin/gh/laithsakka/296/head 2025-10-10T00:37:13.1582199Z * [new branch] gh/laithsakka/296/orig -> origin/gh/laithsakka/296/orig 2025-10-10T00:37:13.1582364Z * [new branch] gh/laithsakka/297/base -> origin/gh/laithsakka/297/base 2025-10-10T00:37:13.1582510Z * [new branch] gh/laithsakka/297/head -> origin/gh/laithsakka/297/head 2025-10-10T00:37:13.1582650Z * [new branch] gh/laithsakka/297/orig -> origin/gh/laithsakka/297/orig 2025-10-10T00:37:13.1582798Z * [new branch] gh/laithsakka/298/base -> origin/gh/laithsakka/298/base 2025-10-10T00:37:13.1582938Z * [new branch] gh/laithsakka/298/head -> origin/gh/laithsakka/298/head 2025-10-10T00:37:13.1583259Z * [new branch] gh/laithsakka/298/orig -> origin/gh/laithsakka/298/orig 2025-10-10T00:37:13.1583411Z * [new branch] gh/laithsakka/299/base -> origin/gh/laithsakka/299/base 2025-10-10T00:37:13.1583557Z * [new branch] gh/laithsakka/299/head -> origin/gh/laithsakka/299/head 2025-10-10T00:37:13.1583725Z * [new branch] gh/laithsakka/299/orig -> origin/gh/laithsakka/299/orig 2025-10-10T00:37:13.1583883Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-10-10T00:37:13.1584040Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-10-10T00:37:13.1584191Z * [new branch] gh/laithsakka/300/base -> origin/gh/laithsakka/300/base 2025-10-10T00:37:13.1584340Z * [new branch] gh/laithsakka/300/head -> origin/gh/laithsakka/300/head 2025-10-10T00:37:13.1584501Z * [new branch] gh/laithsakka/300/orig -> origin/gh/laithsakka/300/orig 2025-10-10T00:37:13.1584649Z * [new branch] gh/laithsakka/301/base -> origin/gh/laithsakka/301/base 2025-10-10T00:37:13.1584804Z * [new branch] gh/laithsakka/301/head -> origin/gh/laithsakka/301/head 2025-10-10T00:37:13.1584953Z * [new branch] gh/laithsakka/301/orig -> origin/gh/laithsakka/301/orig 2025-10-10T00:37:13.1585108Z * [new branch] gh/laithsakka/302/base -> origin/gh/laithsakka/302/base 2025-10-10T00:37:13.1585265Z * [new branch] gh/laithsakka/302/head -> origin/gh/laithsakka/302/head 2025-10-10T00:37:13.1585415Z * [new branch] gh/laithsakka/302/orig -> origin/gh/laithsakka/302/orig 2025-10-10T00:37:13.1585577Z * [new branch] gh/laithsakka/303/base -> origin/gh/laithsakka/303/base 2025-10-10T00:37:13.1585726Z * [new branch] gh/laithsakka/303/head -> origin/gh/laithsakka/303/head 2025-10-10T00:37:13.1585888Z * [new branch] gh/laithsakka/303/orig -> origin/gh/laithsakka/303/orig 2025-10-10T00:37:13.1586040Z * [new branch] gh/laithsakka/304/base -> origin/gh/laithsakka/304/base 2025-10-10T00:37:13.1586189Z * [new branch] gh/laithsakka/304/head -> origin/gh/laithsakka/304/head 2025-10-10T00:37:13.1586518Z * [new branch] gh/laithsakka/304/orig -> origin/gh/laithsakka/304/orig 2025-10-10T00:37:13.1586724Z * [new branch] gh/laithsakka/305/base -> origin/gh/laithsakka/305/base 2025-10-10T00:37:13.1586878Z * [new branch] gh/laithsakka/305/head -> origin/gh/laithsakka/305/head 2025-10-10T00:37:13.1587023Z * [new branch] gh/laithsakka/305/orig -> origin/gh/laithsakka/305/orig 2025-10-10T00:37:13.1587180Z * [new branch] gh/laithsakka/306/base -> origin/gh/laithsakka/306/base 2025-10-10T00:37:13.1587340Z * [new branch] gh/laithsakka/306/head -> origin/gh/laithsakka/306/head 2025-10-10T00:37:13.1587481Z * [new branch] gh/laithsakka/306/orig -> origin/gh/laithsakka/306/orig 2025-10-10T00:37:13.1587627Z * [new branch] gh/laithsakka/307/base -> origin/gh/laithsakka/307/base 2025-10-10T00:37:13.1587764Z * [new branch] gh/laithsakka/307/head -> origin/gh/laithsakka/307/head 2025-10-10T00:37:13.1587921Z * [new branch] gh/laithsakka/307/orig -> origin/gh/laithsakka/307/orig 2025-10-10T00:37:13.1588055Z * [new branch] gh/laithsakka/308/base -> origin/gh/laithsakka/308/base 2025-10-10T00:37:13.1588214Z * [new branch] gh/laithsakka/308/head -> origin/gh/laithsakka/308/head 2025-10-10T00:37:13.1588351Z * [new branch] gh/laithsakka/308/orig -> origin/gh/laithsakka/308/orig 2025-10-10T00:37:13.1588493Z * [new branch] gh/laithsakka/309/base -> origin/gh/laithsakka/309/base 2025-10-10T00:37:13.1588693Z * [new branch] gh/laithsakka/309/head -> origin/gh/laithsakka/309/head 2025-10-10T00:37:13.1588843Z * [new branch] gh/laithsakka/309/orig -> origin/gh/laithsakka/309/orig 2025-10-10T00:37:13.1589000Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-10-10T00:37:13.1589150Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-10-10T00:37:13.1589310Z * [new branch] gh/laithsakka/310/base -> origin/gh/laithsakka/310/base 2025-10-10T00:37:13.1589456Z * [new branch] gh/laithsakka/310/head -> origin/gh/laithsakka/310/head 2025-10-10T00:37:13.1589604Z * [new branch] gh/laithsakka/310/orig -> origin/gh/laithsakka/310/orig 2025-10-10T00:37:13.1589761Z * [new branch] gh/laithsakka/311/base -> origin/gh/laithsakka/311/base 2025-10-10T00:37:13.1589907Z * [new branch] gh/laithsakka/311/head -> origin/gh/laithsakka/311/head 2025-10-10T00:37:13.1590065Z * [new branch] gh/laithsakka/311/orig -> origin/gh/laithsakka/311/orig 2025-10-10T00:37:13.1590213Z * [new branch] gh/laithsakka/312/base -> origin/gh/laithsakka/312/base 2025-10-10T00:37:13.1590360Z * [new branch] gh/laithsakka/312/head -> origin/gh/laithsakka/312/head 2025-10-10T00:37:13.1590513Z * [new branch] gh/laithsakka/312/orig -> origin/gh/laithsakka/312/orig 2025-10-10T00:37:13.1590665Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-10-10T00:37:13.1590819Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-10-10T00:37:13.1590966Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-10-10T00:37:13.1591120Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-10-10T00:37:13.1591268Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-10-10T00:37:13.1591418Z * [new branch] gh/liangel/1/base -> origin/gh/liangel/1/base 2025-10-10T00:37:13.1591567Z * [new branch] gh/liangel/1/head -> origin/gh/liangel/1/head 2025-10-10T00:37:13.1591703Z * [new branch] gh/liangel/1/orig -> origin/gh/liangel/1/orig 2025-10-10T00:37:13.1591902Z * [new branch] gh/liangel/2/base -> origin/gh/liangel/2/base 2025-10-10T00:37:13.1592033Z * [new branch] gh/liangel/2/head -> origin/gh/liangel/2/head 2025-10-10T00:37:13.1592172Z * [new branch] gh/liangel/2/orig -> origin/gh/liangel/2/orig 2025-10-10T00:37:13.1592306Z * [new branch] gh/liangel/3/base -> origin/gh/liangel/3/base 2025-10-10T00:37:13.1592443Z * [new branch] gh/liangel/3/head -> origin/gh/liangel/3/head 2025-10-10T00:37:13.1592587Z * [new branch] gh/liangel/3/orig -> origin/gh/liangel/3/orig 2025-10-10T00:37:13.1592719Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-10-10T00:37:13.1592858Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-10-10T00:37:13.1592991Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-10-10T00:37:13.1597447Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-10-10T00:37:13.1599196Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-10-10T00:37:13.1599522Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-10-10T00:37:13.1599684Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-10-10T00:37:13.1599810Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-10-10T00:37:13.1600216Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-10-10T00:37:13.1600468Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-10-10T00:37:13.1600616Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-10-10T00:37:13.1600787Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-10-10T00:37:13.1600954Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-10-10T00:37:13.1601095Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-10-10T00:37:13.1605353Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-10-10T00:37:13.1605883Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-10-10T00:37:13.1606048Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-10-10T00:37:13.1606194Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-10-10T00:37:13.1606363Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-10-10T00:37:13.1606496Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-10-10T00:37:13.1606920Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-10-10T00:37:13.1608113Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-10-10T00:37:13.1608426Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-10-10T00:37:13.1609382Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-10-10T00:37:13.1610282Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-10-10T00:37:13.1611265Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-10-10T00:37:13.1611618Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-10-10T00:37:13.1612869Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-10-10T00:37:13.1613083Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-10-10T00:37:13.1614132Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-10-10T00:37:13.1614851Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-10-10T00:37:13.1615462Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-10-10T00:37:13.1616394Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-10-10T00:37:13.1617328Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-10-10T00:37:13.1619702Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-10-10T00:37:13.1620037Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-10-10T00:37:13.1620198Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-10-10T00:37:13.1620355Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-10-10T00:37:13.1620514Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-10-10T00:37:13.1622045Z * [new branch] gh/malfet/505/base -> origin/gh/malfet/505/base 2025-10-10T00:37:13.1622302Z * [new branch] gh/malfet/505/head -> origin/gh/malfet/505/head 2025-10-10T00:37:13.1623418Z * [new branch] gh/malfet/505/orig -> origin/gh/malfet/505/orig 2025-10-10T00:37:13.1624230Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-10-10T00:37:13.1624732Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-10-10T00:37:13.1625863Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-10-10T00:37:13.1626854Z * [new branch] gh/malfet/507/base -> origin/gh/malfet/507/base 2025-10-10T00:37:13.1627460Z * [new branch] gh/malfet/507/head -> origin/gh/malfet/507/head 2025-10-10T00:37:13.1629134Z * [new branch] gh/malfet/507/orig -> origin/gh/malfet/507/orig 2025-10-10T00:37:13.1629316Z * [new branch] gh/malfet/513/base -> origin/gh/malfet/513/base 2025-10-10T00:37:13.1629518Z * [new branch] gh/malfet/513/head -> origin/gh/malfet/513/head 2025-10-10T00:37:13.1635266Z * [new branch] gh/malfet/513/orig -> origin/gh/malfet/513/orig 2025-10-10T00:37:13.1638811Z * [new branch] gh/malfet/516/base -> origin/gh/malfet/516/base 2025-10-10T00:37:13.1639085Z * [new branch] gh/malfet/516/head -> origin/gh/malfet/516/head 2025-10-10T00:37:13.1644125Z * [new branch] gh/malfet/516/orig -> origin/gh/malfet/516/orig 2025-10-10T00:37:13.1644310Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-10-10T00:37:13.1644452Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-10-10T00:37:13.1644578Z * [new branch] gh/malfet/518/base -> origin/gh/malfet/518/base 2025-10-10T00:37:13.1644716Z * [new branch] gh/malfet/518/head -> origin/gh/malfet/518/head 2025-10-10T00:37:13.1644852Z * [new branch] gh/malfet/518/orig -> origin/gh/malfet/518/orig 2025-10-10T00:37:13.1644984Z * [new branch] gh/malfet/519/base -> origin/gh/malfet/519/base 2025-10-10T00:37:13.1645124Z * [new branch] gh/malfet/519/head -> origin/gh/malfet/519/head 2025-10-10T00:37:13.1645253Z * [new branch] gh/malfet/519/orig -> origin/gh/malfet/519/orig 2025-10-10T00:37:13.1645380Z * [new branch] gh/malfet/520/base -> origin/gh/malfet/520/base 2025-10-10T00:37:13.1645513Z * [new branch] gh/malfet/520/head -> origin/gh/malfet/520/head 2025-10-10T00:37:13.1645638Z * [new branch] gh/malfet/520/orig -> origin/gh/malfet/520/orig 2025-10-10T00:37:13.1645774Z * [new branch] gh/malfet/521/base -> origin/gh/malfet/521/base 2025-10-10T00:37:13.1646099Z * [new branch] gh/malfet/521/head -> origin/gh/malfet/521/head 2025-10-10T00:37:13.1646232Z * [new branch] gh/malfet/521/orig -> origin/gh/malfet/521/orig 2025-10-10T00:37:13.1646364Z * [new branch] gh/malfet/522/base -> origin/gh/malfet/522/base 2025-10-10T00:37:13.1646492Z * [new branch] gh/malfet/522/head -> origin/gh/malfet/522/head 2025-10-10T00:37:13.1646631Z * [new branch] gh/malfet/522/orig -> origin/gh/malfet/522/orig 2025-10-10T00:37:13.1647140Z * [new branch] gh/malfet/523/base -> origin/gh/malfet/523/base 2025-10-10T00:37:13.1647696Z * [new branch] gh/malfet/523/head -> origin/gh/malfet/523/head 2025-10-10T00:37:13.1648618Z * [new branch] gh/malfet/523/orig -> origin/gh/malfet/523/orig 2025-10-10T00:37:13.1649804Z * [new branch] gh/malfet/524/base -> origin/gh/malfet/524/base 2025-10-10T00:37:13.1649950Z * [new branch] gh/malfet/524/head -> origin/gh/malfet/524/head 2025-10-10T00:37:13.1650616Z * [new branch] gh/malfet/524/orig -> origin/gh/malfet/524/orig 2025-10-10T00:37:13.1654820Z * [new branch] gh/malfet/525/base -> origin/gh/malfet/525/base 2025-10-10T00:37:13.1654993Z * [new branch] gh/malfet/525/head -> origin/gh/malfet/525/head 2025-10-10T00:37:13.1655125Z * [new branch] gh/malfet/525/orig -> origin/gh/malfet/525/orig 2025-10-10T00:37:13.1655441Z * [new branch] gh/malfet/526/base -> origin/gh/malfet/526/base 2025-10-10T00:37:13.1655586Z * [new branch] gh/malfet/526/head -> origin/gh/malfet/526/head 2025-10-10T00:37:13.1655717Z * [new branch] gh/malfet/526/orig -> origin/gh/malfet/526/orig 2025-10-10T00:37:13.1656337Z * [new branch] gh/malfet/527/base -> origin/gh/malfet/527/base 2025-10-10T00:37:13.1656510Z * [new branch] gh/malfet/527/head -> origin/gh/malfet/527/head 2025-10-10T00:37:13.1657005Z * [new branch] gh/malfet/527/orig -> origin/gh/malfet/527/orig 2025-10-10T00:37:13.1658258Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-10-10T00:37:13.1661625Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-10-10T00:37:13.1661820Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-10-10T00:37:13.1661991Z * [new branch] gh/malfet/529/base -> origin/gh/malfet/529/base 2025-10-10T00:37:13.1662185Z * [new branch] gh/malfet/529/head -> origin/gh/malfet/529/head 2025-10-10T00:37:13.1665159Z * [new branch] gh/malfet/529/orig -> origin/gh/malfet/529/orig 2025-10-10T00:37:13.1665485Z * [new branch] gh/malfet/530/base -> origin/gh/malfet/530/base 2025-10-10T00:37:13.1665653Z * [new branch] gh/malfet/530/head -> origin/gh/malfet/530/head 2025-10-10T00:37:13.1665800Z * [new branch] gh/malfet/530/orig -> origin/gh/malfet/530/orig 2025-10-10T00:37:13.1665942Z * [new branch] gh/malfet/531/base -> origin/gh/malfet/531/base 2025-10-10T00:37:13.1666084Z * [new branch] gh/malfet/531/head -> origin/gh/malfet/531/head 2025-10-10T00:37:13.1666230Z * [new branch] gh/malfet/531/orig -> origin/gh/malfet/531/orig 2025-10-10T00:37:13.1666714Z * [new branch] gh/malfet/532/base -> origin/gh/malfet/532/base 2025-10-10T00:37:13.1675338Z * [new branch] gh/malfet/532/head -> origin/gh/malfet/532/head 2025-10-10T00:37:13.1680324Z * [new branch] gh/malfet/532/orig -> origin/gh/malfet/532/orig 2025-10-10T00:37:13.1682430Z * [new branch] gh/malfet/533/base -> origin/gh/malfet/533/base 2025-10-10T00:37:13.1682781Z * [new branch] gh/malfet/533/head -> origin/gh/malfet/533/head 2025-10-10T00:37:13.1682923Z * [new branch] gh/malfet/533/orig -> origin/gh/malfet/533/orig 2025-10-10T00:37:13.1683053Z * [new branch] gh/malfet/534/base -> origin/gh/malfet/534/base 2025-10-10T00:37:13.1683186Z * [new branch] gh/malfet/534/head -> origin/gh/malfet/534/head 2025-10-10T00:37:13.1683313Z * [new branch] gh/malfet/534/orig -> origin/gh/malfet/534/orig 2025-10-10T00:37:13.1683443Z * [new branch] gh/malfet/535/base -> origin/gh/malfet/535/base 2025-10-10T00:37:13.1683577Z * [new branch] gh/malfet/535/head -> origin/gh/malfet/535/head 2025-10-10T00:37:13.1683700Z * [new branch] gh/malfet/535/orig -> origin/gh/malfet/535/orig 2025-10-10T00:37:13.1683830Z * [new branch] gh/malfet/536/base -> origin/gh/malfet/536/base 2025-10-10T00:37:13.1683955Z * [new branch] gh/malfet/536/head -> origin/gh/malfet/536/head 2025-10-10T00:37:13.1684079Z * [new branch] gh/malfet/536/orig -> origin/gh/malfet/536/orig 2025-10-10T00:37:13.1684211Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-10-10T00:37:13.1684335Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-10-10T00:37:13.1684465Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-10-10T00:37:13.1684628Z * [new branch] gh/malfet/538/base -> origin/gh/malfet/538/base 2025-10-10T00:37:13.1684763Z * [new branch] gh/malfet/538/head -> origin/gh/malfet/538/head 2025-10-10T00:37:13.1684885Z * [new branch] gh/malfet/538/orig -> origin/gh/malfet/538/orig 2025-10-10T00:37:13.1685007Z * [new branch] gh/malfet/539/base -> origin/gh/malfet/539/base 2025-10-10T00:37:13.1690012Z * [new branch] gh/malfet/539/head -> origin/gh/malfet/539/head 2025-10-10T00:37:13.1694852Z * [new branch] gh/malfet/539/orig -> origin/gh/malfet/539/orig 2025-10-10T00:37:13.1695028Z * [new branch] gh/malfet/540/base -> origin/gh/malfet/540/base 2025-10-10T00:37:13.1695178Z * [new branch] gh/malfet/540/head -> origin/gh/malfet/540/head 2025-10-10T00:37:13.1695335Z * [new branch] gh/malfet/540/orig -> origin/gh/malfet/540/orig 2025-10-10T00:37:13.1695466Z * [new branch] gh/malfet/541/base -> origin/gh/malfet/541/base 2025-10-10T00:37:13.1695604Z * [new branch] gh/malfet/541/head -> origin/gh/malfet/541/head 2025-10-10T00:37:13.1695735Z * [new branch] gh/malfet/541/orig -> origin/gh/malfet/541/orig 2025-10-10T00:37:13.1695869Z * [new branch] gh/malfet/542/base -> origin/gh/malfet/542/base 2025-10-10T00:37:13.1696010Z * [new branch] gh/malfet/542/head -> origin/gh/malfet/542/head 2025-10-10T00:37:13.1696137Z * [new branch] gh/malfet/542/orig -> origin/gh/malfet/542/orig 2025-10-10T00:37:13.1696270Z * [new branch] gh/malfet/543/base -> origin/gh/malfet/543/base 2025-10-10T00:37:13.1696397Z * [new branch] gh/malfet/543/head -> origin/gh/malfet/543/head 2025-10-10T00:37:13.1696533Z * [new branch] gh/malfet/543/orig -> origin/gh/malfet/543/orig 2025-10-10T00:37:13.1696659Z * [new branch] gh/malfet/544/base -> origin/gh/malfet/544/base 2025-10-10T00:37:13.1696782Z * [new branch] gh/malfet/544/head -> origin/gh/malfet/544/head 2025-10-10T00:37:13.1696914Z * [new branch] gh/malfet/544/orig -> origin/gh/malfet/544/orig 2025-10-10T00:37:13.1697179Z * [new branch] gh/malfet/545/base -> origin/gh/malfet/545/base 2025-10-10T00:37:13.1697314Z * [new branch] gh/malfet/545/head -> origin/gh/malfet/545/head 2025-10-10T00:37:13.1697443Z * [new branch] gh/malfet/545/orig -> origin/gh/malfet/545/orig 2025-10-10T00:37:13.1701652Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-10-10T00:37:13.1702001Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-10-10T00:37:13.1702163Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-10-10T00:37:13.1702308Z * [new branch] gh/malfet/547/base -> origin/gh/malfet/547/base 2025-10-10T00:37:13.1702445Z * [new branch] gh/malfet/547/head -> origin/gh/malfet/547/head 2025-10-10T00:37:13.1702588Z * [new branch] gh/malfet/547/orig -> origin/gh/malfet/547/orig 2025-10-10T00:37:13.1702738Z * [new branch] gh/malfet/548/base -> origin/gh/malfet/548/base 2025-10-10T00:37:13.1702882Z * [new branch] gh/malfet/548/head -> origin/gh/malfet/548/head 2025-10-10T00:37:13.1703017Z * [new branch] gh/malfet/548/orig -> origin/gh/malfet/548/orig 2025-10-10T00:37:13.1703152Z * [new branch] gh/malfet/549/base -> origin/gh/malfet/549/base 2025-10-10T00:37:13.1703293Z * [new branch] gh/malfet/549/head -> origin/gh/malfet/549/head 2025-10-10T00:37:13.1703562Z * [new branch] gh/malfet/549/orig -> origin/gh/malfet/549/orig 2025-10-10T00:37:13.1704078Z * [new branch] gh/malfet/550/base -> origin/gh/malfet/550/base 2025-10-10T00:37:13.1704259Z * [new branch] gh/malfet/550/head -> origin/gh/malfet/550/head 2025-10-10T00:37:13.1705278Z * [new branch] gh/malfet/550/orig -> origin/gh/malfet/550/orig 2025-10-10T00:37:13.1705910Z * [new branch] gh/malfet/551/base -> origin/gh/malfet/551/base 2025-10-10T00:37:13.1706942Z * [new branch] gh/malfet/551/head -> origin/gh/malfet/551/head 2025-10-10T00:37:13.1707104Z * [new branch] gh/malfet/551/orig -> origin/gh/malfet/551/orig 2025-10-10T00:37:13.1710602Z * [new branch] gh/malfet/552/base -> origin/gh/malfet/552/base 2025-10-10T00:37:13.1710774Z * [new branch] gh/malfet/552/head -> origin/gh/malfet/552/head 2025-10-10T00:37:13.1711672Z * [new branch] gh/malfet/552/orig -> origin/gh/malfet/552/orig 2025-10-10T00:37:13.1712297Z * [new branch] gh/malfet/553/base -> origin/gh/malfet/553/base 2025-10-10T00:37:13.1712476Z * [new branch] gh/malfet/553/head -> origin/gh/malfet/553/head 2025-10-10T00:37:13.1712867Z * [new branch] gh/malfet/553/orig -> origin/gh/malfet/553/orig 2025-10-10T00:37:13.1716683Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-10-10T00:37:13.1716934Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-10-10T00:37:13.1717116Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-10-10T00:37:13.1717290Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-10-10T00:37:13.1717448Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-10-10T00:37:13.1717913Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-10-10T00:37:13.1718096Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-10-10T00:37:13.1719265Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-10-10T00:37:13.1719837Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-10-10T00:37:13.1720499Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-10-10T00:37:13.1721451Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-10-10T00:37:13.1722749Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-10-10T00:37:13.1723902Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-10-10T00:37:13.1724440Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-10-10T00:37:13.1725700Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-10-10T00:37:13.1726722Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-10-10T00:37:13.1727251Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-10-10T00:37:13.1728050Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-10-10T00:37:13.1729134Z * [new branch] gh/masnesral/237/base -> origin/gh/masnesral/237/base 2025-10-10T00:37:13.1729861Z * [new branch] gh/masnesral/237/head -> origin/gh/masnesral/237/head 2025-10-10T00:37:13.1730858Z * [new branch] gh/masnesral/237/orig -> origin/gh/masnesral/237/orig 2025-10-10T00:37:13.1731782Z * [new branch] gh/masnesral/238/base -> origin/gh/masnesral/238/base 2025-10-10T00:37:13.1734540Z * [new branch] gh/masnesral/238/head -> origin/gh/masnesral/238/head 2025-10-10T00:37:13.1734730Z * [new branch] gh/masnesral/238/orig -> origin/gh/masnesral/238/orig 2025-10-10T00:37:13.1734890Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-10-10T00:37:13.1735282Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-10-10T00:37:13.1735745Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-10-10T00:37:13.1739719Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-10-10T00:37:13.1739894Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-10-10T00:37:13.1740041Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-10-10T00:37:13.1740179Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-10-10T00:37:13.1740348Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-10-10T00:37:13.1740678Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-10-10T00:37:13.1740993Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-10-10T00:37:13.1746685Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-10-10T00:37:13.1746921Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-10-10T00:37:13.1747078Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-10-10T00:37:13.1747237Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-10-10T00:37:13.1747448Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-10-10T00:37:13.1747639Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-10-10T00:37:13.1747839Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-10-10T00:37:13.1748039Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-10-10T00:37:13.1748418Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-10-10T00:37:13.1748964Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-10-10T00:37:13.1751491Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-10-10T00:37:13.1751866Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-10-10T00:37:13.1752145Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-10-10T00:37:13.1752347Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-10-10T00:37:13.1753778Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-10-10T00:37:13.1754094Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-10-10T00:37:13.1754508Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-10-10T00:37:13.1756877Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-10-10T00:37:13.1757280Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-10-10T00:37:13.1757599Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-10-10T00:37:13.1762263Z * [new branch] gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base 2025-10-10T00:37:13.1762615Z * [new branch] gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head 2025-10-10T00:37:13.1763023Z * [new branch] gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig 2025-10-10T00:37:13.1763325Z * [new branch] gh/mikaylagawarecki/340/base -> origin/gh/mikaylagawarecki/340/base 2025-10-10T00:37:13.1763566Z * [new branch] gh/mikaylagawarecki/340/head -> origin/gh/mikaylagawarecki/340/head 2025-10-10T00:37:13.1763743Z * [new branch] gh/mikaylagawarecki/340/orig -> origin/gh/mikaylagawarecki/340/orig 2025-10-10T00:37:13.1764004Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-10-10T00:37:13.1764618Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-10-10T00:37:13.1764823Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-10-10T00:37:13.1768995Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-10-10T00:37:13.1769227Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-10-10T00:37:13.1769402Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-10-10T00:37:13.1769577Z * [new branch] gh/mikaylagawarecki/343/base -> origin/gh/mikaylagawarecki/343/base 2025-10-10T00:37:13.1769742Z * [new branch] gh/mikaylagawarecki/343/head -> origin/gh/mikaylagawarecki/343/head 2025-10-10T00:37:13.1770065Z * [new branch] gh/mikaylagawarecki/343/orig -> origin/gh/mikaylagawarecki/343/orig 2025-10-10T00:37:13.1770516Z * [new branch] gh/mikaylagawarecki/344/base -> origin/gh/mikaylagawarecki/344/base 2025-10-10T00:37:13.1772201Z * [new branch] gh/mikaylagawarecki/344/head -> origin/gh/mikaylagawarecki/344/head 2025-10-10T00:37:13.1772546Z * [new branch] gh/mikaylagawarecki/344/orig -> origin/gh/mikaylagawarecki/344/orig 2025-10-10T00:37:13.1773054Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-10-10T00:37:13.1775175Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-10-10T00:37:13.1775520Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-10-10T00:37:13.1775764Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-10-10T00:37:13.1776191Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-10-10T00:37:13.1780330Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-10-10T00:37:13.1780711Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-10-10T00:37:13.1780982Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-10-10T00:37:13.1781221Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-10-10T00:37:13.1781488Z * [new branch] gh/mikaylagawarecki/348/base -> origin/gh/mikaylagawarecki/348/base 2025-10-10T00:37:13.1781686Z * [new branch] gh/mikaylagawarecki/348/head -> origin/gh/mikaylagawarecki/348/head 2025-10-10T00:37:13.1782431Z * [new branch] gh/mikaylagawarecki/348/orig -> origin/gh/mikaylagawarecki/348/orig 2025-10-10T00:37:13.1782833Z * [new branch] gh/mikaylagawarecki/349/base -> origin/gh/mikaylagawarecki/349/base 2025-10-10T00:37:13.1786171Z * [new branch] gh/mikaylagawarecki/349/head -> origin/gh/mikaylagawarecki/349/head 2025-10-10T00:37:13.1786656Z * [new branch] gh/mikaylagawarecki/349/orig -> origin/gh/mikaylagawarecki/349/orig 2025-10-10T00:37:13.1786845Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-10-10T00:37:13.1787167Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-10-10T00:37:13.1787490Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-10-10T00:37:13.1790010Z * [new branch] gh/mlazos/18/base -> origin/gh/mlazos/18/base 2025-10-10T00:37:13.1796517Z * [new branch] gh/mlazos/18/head -> origin/gh/mlazos/18/head 2025-10-10T00:37:13.1801738Z * [new branch] gh/mlazos/18/orig -> origin/gh/mlazos/18/orig 2025-10-10T00:37:13.1801919Z * [new branch] gh/mlazos/19/base -> origin/gh/mlazos/19/base 2025-10-10T00:37:13.1802064Z * [new branch] gh/mlazos/19/head -> origin/gh/mlazos/19/head 2025-10-10T00:37:13.1802202Z * [new branch] gh/mlazos/19/orig -> origin/gh/mlazos/19/orig 2025-10-10T00:37:13.1802360Z * [new branch] gh/mlazos/20/base -> origin/gh/mlazos/20/base 2025-10-10T00:37:13.1802511Z * [new branch] gh/mlazos/20/head -> origin/gh/mlazos/20/head 2025-10-10T00:37:13.1802656Z * [new branch] gh/mlazos/20/orig -> origin/gh/mlazos/20/orig 2025-10-10T00:37:13.1802792Z * [new branch] gh/mlazos/21/base -> origin/gh/mlazos/21/base 2025-10-10T00:37:13.1802980Z * [new branch] gh/mlazos/21/head -> origin/gh/mlazos/21/head 2025-10-10T00:37:13.1803123Z * [new branch] gh/mlazos/21/orig -> origin/gh/mlazos/21/orig 2025-10-10T00:37:13.1803255Z * [new branch] gh/mlazos/22/base -> origin/gh/mlazos/22/base 2025-10-10T00:37:13.1803394Z * [new branch] gh/mlazos/22/head -> origin/gh/mlazos/22/head 2025-10-10T00:37:13.1803527Z * [new branch] gh/mlazos/22/orig -> origin/gh/mlazos/22/orig 2025-10-10T00:37:13.1803667Z * [new branch] gh/mlazos/23/base -> origin/gh/mlazos/23/base 2025-10-10T00:37:13.1803804Z * [new branch] gh/mlazos/23/head -> origin/gh/mlazos/23/head 2025-10-10T00:37:13.1803942Z * [new branch] gh/mlazos/23/orig -> origin/gh/mlazos/23/orig 2025-10-10T00:37:13.1804075Z * [new branch] gh/mlazos/24/base -> origin/gh/mlazos/24/base 2025-10-10T00:37:13.1804207Z * [new branch] gh/mlazos/24/head -> origin/gh/mlazos/24/head 2025-10-10T00:37:13.1804530Z * [new branch] gh/mlazos/24/orig -> origin/gh/mlazos/24/orig 2025-10-10T00:37:13.1808790Z * [new branch] gh/mlazos/25/base -> origin/gh/mlazos/25/base 2025-10-10T00:37:13.1809044Z * [new branch] gh/mlazos/25/head -> origin/gh/mlazos/25/head 2025-10-10T00:37:13.1809197Z * [new branch] gh/mlazos/25/orig -> origin/gh/mlazos/25/orig 2025-10-10T00:37:13.1809422Z * [new branch] gh/mlazos/26/base -> origin/gh/mlazos/26/base 2025-10-10T00:37:13.1809581Z * [new branch] gh/mlazos/26/head -> origin/gh/mlazos/26/head 2025-10-10T00:37:13.1809812Z * [new branch] gh/mlazos/26/orig -> origin/gh/mlazos/26/orig 2025-10-10T00:37:13.1809960Z * [new branch] gh/mlazos/27/base -> origin/gh/mlazos/27/base 2025-10-10T00:37:13.1814635Z * [new branch] gh/mlazos/27/head -> origin/gh/mlazos/27/head 2025-10-10T00:37:13.1814830Z * [new branch] gh/mlazos/27/orig -> origin/gh/mlazos/27/orig 2025-10-10T00:37:13.1815094Z * [new branch] gh/mlazos/28/base -> origin/gh/mlazos/28/base 2025-10-10T00:37:13.1815240Z * [new branch] gh/mlazos/28/head -> origin/gh/mlazos/28/head 2025-10-10T00:37:13.1815380Z * [new branch] gh/mlazos/28/orig -> origin/gh/mlazos/28/orig 2025-10-10T00:37:13.1818975Z * [new branch] gh/mlazos/29/base -> origin/gh/mlazos/29/base 2025-10-10T00:37:13.1819730Z * [new branch] gh/mlazos/29/head -> origin/gh/mlazos/29/head 2025-10-10T00:37:13.1819914Z * [new branch] gh/mlazos/29/orig -> origin/gh/mlazos/29/orig 2025-10-10T00:37:13.1820114Z * [new branch] gh/mlazos/30/base -> origin/gh/mlazos/30/base 2025-10-10T00:37:13.1820262Z * [new branch] gh/mlazos/30/head -> origin/gh/mlazos/30/head 2025-10-10T00:37:13.1820414Z * [new branch] gh/mlazos/30/orig -> origin/gh/mlazos/30/orig 2025-10-10T00:37:13.1820562Z * [new branch] gh/mlazos/31/base -> origin/gh/mlazos/31/base 2025-10-10T00:37:13.1820710Z * [new branch] gh/mlazos/31/head -> origin/gh/mlazos/31/head 2025-10-10T00:37:13.1820858Z * [new branch] gh/mlazos/31/orig -> origin/gh/mlazos/31/orig 2025-10-10T00:37:13.1821004Z * [new branch] gh/mlazos/32/base -> origin/gh/mlazos/32/base 2025-10-10T00:37:13.1821651Z * [new branch] gh/mlazos/32/head -> origin/gh/mlazos/32/head 2025-10-10T00:37:13.1821805Z * [new branch] gh/mlazos/32/orig -> origin/gh/mlazos/32/orig 2025-10-10T00:37:13.1822058Z * [new branch] gh/mlazos/33/base -> origin/gh/mlazos/33/base 2025-10-10T00:37:13.1822216Z * [new branch] gh/mlazos/33/head -> origin/gh/mlazos/33/head 2025-10-10T00:37:13.1822367Z * [new branch] gh/mlazos/33/orig -> origin/gh/mlazos/33/orig 2025-10-10T00:37:13.1823442Z * [new branch] gh/mlazos/34/base -> origin/gh/mlazos/34/base 2025-10-10T00:37:13.1824011Z * [new branch] gh/mlazos/34/head -> origin/gh/mlazos/34/head 2025-10-10T00:37:13.1824725Z * [new branch] gh/mlazos/34/orig -> origin/gh/mlazos/34/orig 2025-10-10T00:37:13.1825754Z * [new branch] gh/mlazos/35/base -> origin/gh/mlazos/35/base 2025-10-10T00:37:13.1826109Z * [new branch] gh/mlazos/35/head -> origin/gh/mlazos/35/head 2025-10-10T00:37:13.1827256Z * [new branch] gh/mlazos/35/orig -> origin/gh/mlazos/35/orig 2025-10-10T00:37:13.1832268Z * [new branch] gh/mlazos/36/base -> origin/gh/mlazos/36/base 2025-10-10T00:37:13.1832410Z * [new branch] gh/mlazos/36/head -> origin/gh/mlazos/36/head 2025-10-10T00:37:13.1832733Z * [new branch] gh/mlazos/36/orig -> origin/gh/mlazos/36/orig 2025-10-10T00:37:13.1832878Z * [new branch] gh/mlazos/37/base -> origin/gh/mlazos/37/base 2025-10-10T00:37:13.1833012Z * [new branch] gh/mlazos/37/head -> origin/gh/mlazos/37/head 2025-10-10T00:37:13.1833158Z * [new branch] gh/mlazos/37/orig -> origin/gh/mlazos/37/orig 2025-10-10T00:37:13.1838416Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-10-10T00:37:13.1838625Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-10-10T00:37:13.1838806Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-10-10T00:37:13.1838981Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-10-10T00:37:13.1839160Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-10-10T00:37:13.1839339Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-10-10T00:37:13.1842831Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-10-10T00:37:13.1843008Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-10-10T00:37:13.1843168Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-10-10T00:37:13.1843458Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-10-10T00:37:13.1844021Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-10-10T00:37:13.1844310Z * [new branch] gh/muchulee8/66/base -> origin/gh/muchulee8/66/base 2025-10-10T00:37:13.1852828Z * [new branch] gh/muchulee8/66/head -> origin/gh/muchulee8/66/head 2025-10-10T00:37:13.1853021Z * [new branch] gh/muchulee8/66/orig -> origin/gh/muchulee8/66/orig 2025-10-10T00:37:13.1853218Z * [new branch] gh/muchulee8/67/base -> origin/gh/muchulee8/67/base 2025-10-10T00:37:13.1853358Z * [new branch] gh/muchulee8/67/head -> origin/gh/muchulee8/67/head 2025-10-10T00:37:13.1853505Z * [new branch] gh/muchulee8/67/orig -> origin/gh/muchulee8/67/orig 2025-10-10T00:37:13.1853678Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-10-10T00:37:13.1853865Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-10-10T00:37:13.1854016Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-10-10T00:37:13.1854175Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-10-10T00:37:13.1854322Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-10-10T00:37:13.1854475Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-10-10T00:37:13.1854631Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-10-10T00:37:13.1854774Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-10-10T00:37:13.1854924Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-10-10T00:37:13.1858957Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-10-10T00:37:13.1859169Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-10-10T00:37:13.1859321Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-10-10T00:37:13.1859502Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-10-10T00:37:13.1860788Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-10-10T00:37:13.1861111Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-10-10T00:37:13.1861288Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-10-10T00:37:13.1861451Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-10-10T00:37:13.1861627Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-10-10T00:37:13.1861808Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-10-10T00:37:13.1861965Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-10-10T00:37:13.1862232Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-10-10T00:37:13.1863251Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-10-10T00:37:13.1863605Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-10-10T00:37:13.1864681Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-10-10T00:37:13.1866594Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-10-10T00:37:13.1866896Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-10-10T00:37:13.1867051Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-10-10T00:37:13.1872291Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-10-10T00:37:13.1876864Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-10-10T00:37:13.1881994Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-10-10T00:37:13.1882179Z * [new branch] gh/nikitaved/3/base -> origin/gh/nikitaved/3/base 2025-10-10T00:37:13.1882349Z * [new branch] gh/nikitaved/3/head -> origin/gh/nikitaved/3/head 2025-10-10T00:37:13.1882489Z * [new branch] gh/nikitaved/3/orig -> origin/gh/nikitaved/3/orig 2025-10-10T00:37:13.1882631Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-10-10T00:37:13.1882772Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-10-10T00:37:13.1882906Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-10-10T00:37:13.1883115Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-10-10T00:37:13.1883256Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-10-10T00:37:13.1883385Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-10-10T00:37:13.1883522Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-10-10T00:37:13.1883654Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-10-10T00:37:13.1883789Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-10-10T00:37:13.1883916Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-10-10T00:37:13.1884052Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-10-10T00:37:13.1887696Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-10-10T00:37:13.1888076Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-10-10T00:37:13.1888300Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-10-10T00:37:13.1888455Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-10-10T00:37:13.1888586Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-10-10T00:37:13.1889003Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-10-10T00:37:13.1889659Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-10-10T00:37:13.1889831Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-10-10T00:37:13.1890011Z * [new branch] gh/pearu/113/base -> origin/gh/pearu/113/base 2025-10-10T00:37:13.1890143Z * [new branch] gh/pearu/113/head -> origin/gh/pearu/113/head 2025-10-10T00:37:13.1890299Z * [new branch] gh/pearu/113/orig -> origin/gh/pearu/113/orig 2025-10-10T00:37:13.1890432Z * [new branch] gh/pearu/114/base -> origin/gh/pearu/114/base 2025-10-10T00:37:13.1890571Z * [new branch] gh/pearu/114/head -> origin/gh/pearu/114/head 2025-10-10T00:37:13.1890711Z * [new branch] gh/pearu/114/orig -> origin/gh/pearu/114/orig 2025-10-10T00:37:13.1895161Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-10-10T00:37:13.1895333Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-10-10T00:37:13.1895468Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-10-10T00:37:13.1895608Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-10-10T00:37:13.1895736Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-10-10T00:37:13.1896025Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-10-10T00:37:13.1896639Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-10-10T00:37:13.1897190Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-10-10T00:37:13.1901706Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-10-10T00:37:13.1901887Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-10-10T00:37:13.1902037Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-10-10T00:37:13.1902170Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-10-10T00:37:13.1902310Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-10-10T00:37:13.1902439Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-10-10T00:37:13.1904923Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-10-10T00:37:13.1905068Z * [new branch] gh/pearu/120/base -> origin/gh/pearu/120/base 2025-10-10T00:37:13.1905222Z * [new branch] gh/pearu/120/head -> origin/gh/pearu/120/head 2025-10-10T00:37:13.1905369Z * [new branch] gh/pearu/120/orig -> origin/gh/pearu/120/orig 2025-10-10T00:37:13.1905515Z * [new branch] gh/pearu/121/base -> origin/gh/pearu/121/base 2025-10-10T00:37:13.1905916Z * [new branch] gh/pearu/121/head -> origin/gh/pearu/121/head 2025-10-10T00:37:13.1906717Z * [new branch] gh/pearu/121/orig -> origin/gh/pearu/121/orig 2025-10-10T00:37:13.1910903Z * [new branch] gh/pearu/122/base -> origin/gh/pearu/122/base 2025-10-10T00:37:13.1911216Z * [new branch] gh/pearu/122/head -> origin/gh/pearu/122/head 2025-10-10T00:37:13.1911387Z * [new branch] gh/pearu/122/orig -> origin/gh/pearu/122/orig 2025-10-10T00:37:13.1911536Z * [new branch] gh/pearu/123/base -> origin/gh/pearu/123/base 2025-10-10T00:37:13.1911680Z * [new branch] gh/pearu/123/head -> origin/gh/pearu/123/head 2025-10-10T00:37:13.1916638Z * [new branch] gh/pearu/123/orig -> origin/gh/pearu/123/orig 2025-10-10T00:37:13.1917027Z * [new branch] gh/pearu/124/base -> origin/gh/pearu/124/base 2025-10-10T00:37:13.1917183Z * [new branch] gh/pearu/124/head -> origin/gh/pearu/124/head 2025-10-10T00:37:13.1917335Z * [new branch] gh/pearu/124/orig -> origin/gh/pearu/124/orig 2025-10-10T00:37:13.1917496Z * [new branch] gh/pearu/125/base -> origin/gh/pearu/125/base 2025-10-10T00:37:13.1919167Z * [new branch] gh/pearu/125/head -> origin/gh/pearu/125/head 2025-10-10T00:37:13.1919345Z * [new branch] gh/pearu/125/orig -> origin/gh/pearu/125/orig 2025-10-10T00:37:13.1919490Z * [new branch] gh/pearu/126/base -> origin/gh/pearu/126/base 2025-10-10T00:37:13.1919636Z * [new branch] gh/pearu/126/head -> origin/gh/pearu/126/head 2025-10-10T00:37:13.1919778Z * [new branch] gh/pearu/126/orig -> origin/gh/pearu/126/orig 2025-10-10T00:37:13.1919927Z * [new branch] gh/pearu/127/base -> origin/gh/pearu/127/base 2025-10-10T00:37:13.1925323Z * [new branch] gh/pearu/127/head -> origin/gh/pearu/127/head 2025-10-10T00:37:13.1925506Z * [new branch] gh/pearu/127/orig -> origin/gh/pearu/127/orig 2025-10-10T00:37:13.1925670Z * [new branch] gh/pearu/128/base -> origin/gh/pearu/128/base 2025-10-10T00:37:13.1925818Z * [new branch] gh/pearu/128/head -> origin/gh/pearu/128/head 2025-10-10T00:37:13.1926137Z * [new branch] gh/pearu/128/orig -> origin/gh/pearu/128/orig 2025-10-10T00:37:13.1926277Z * [new branch] gh/pearu/129/base -> origin/gh/pearu/129/base 2025-10-10T00:37:13.1926424Z * [new branch] gh/pearu/129/head -> origin/gh/pearu/129/head 2025-10-10T00:37:13.1927531Z * [new branch] gh/pearu/129/orig -> origin/gh/pearu/129/orig 2025-10-10T00:37:13.1927765Z * [new branch] gh/pearu/130/base -> origin/gh/pearu/130/base 2025-10-10T00:37:13.1927897Z * [new branch] gh/pearu/130/head -> origin/gh/pearu/130/head 2025-10-10T00:37:13.1928024Z * [new branch] gh/pearu/130/orig -> origin/gh/pearu/130/orig 2025-10-10T00:37:13.1928160Z * [new branch] gh/pearu/131/base -> origin/gh/pearu/131/base 2025-10-10T00:37:13.1928302Z * [new branch] gh/pearu/131/head -> origin/gh/pearu/131/head 2025-10-10T00:37:13.1936484Z * [new branch] gh/pearu/131/orig -> origin/gh/pearu/131/orig 2025-10-10T00:37:13.1938407Z * [new branch] gh/pearu/132/base -> origin/gh/pearu/132/base 2025-10-10T00:37:13.1938725Z * [new branch] gh/pearu/132/head -> origin/gh/pearu/132/head 2025-10-10T00:37:13.1938889Z * [new branch] gh/pearu/132/orig -> origin/gh/pearu/132/orig 2025-10-10T00:37:13.1939051Z * [new branch] gh/pearu/133/base -> origin/gh/pearu/133/base 2025-10-10T00:37:13.1939186Z * [new branch] gh/pearu/133/head -> origin/gh/pearu/133/head 2025-10-10T00:37:13.1939325Z * [new branch] gh/pearu/133/orig -> origin/gh/pearu/133/orig 2025-10-10T00:37:13.1939584Z * [new branch] gh/pearu/134/base -> origin/gh/pearu/134/base 2025-10-10T00:37:13.1939750Z * [new branch] gh/pearu/134/head -> origin/gh/pearu/134/head 2025-10-10T00:37:13.1939891Z * [new branch] gh/pearu/134/orig -> origin/gh/pearu/134/orig 2025-10-10T00:37:13.1940020Z * [new branch] gh/pearu/135/base -> origin/gh/pearu/135/base 2025-10-10T00:37:13.1942804Z * [new branch] gh/pearu/135/head -> origin/gh/pearu/135/head 2025-10-10T00:37:13.1943126Z * [new branch] gh/pearu/135/orig -> origin/gh/pearu/135/orig 2025-10-10T00:37:13.1943608Z * [new branch] gh/pearu/136/base -> origin/gh/pearu/136/base 2025-10-10T00:37:13.1943781Z * [new branch] gh/pearu/136/head -> origin/gh/pearu/136/head 2025-10-10T00:37:13.1944499Z * [new branch] gh/pearu/136/orig -> origin/gh/pearu/136/orig 2025-10-10T00:37:13.1944679Z * [new branch] gh/pearu/137/base -> origin/gh/pearu/137/base 2025-10-10T00:37:13.1944969Z * [new branch] gh/pearu/137/head -> origin/gh/pearu/137/head 2025-10-10T00:37:13.1950912Z * [new branch] gh/pearu/137/orig -> origin/gh/pearu/137/orig 2025-10-10T00:37:13.1951225Z * [new branch] gh/pearu/138/base -> origin/gh/pearu/138/base 2025-10-10T00:37:13.1951448Z * [new branch] gh/pearu/138/head -> origin/gh/pearu/138/head 2025-10-10T00:37:13.1951618Z * [new branch] gh/pearu/138/orig -> origin/gh/pearu/138/orig 2025-10-10T00:37:13.1951829Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-10-10T00:37:13.1951976Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-10-10T00:37:13.1952686Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-10-10T00:37:13.1952860Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-10-10T00:37:13.1953008Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-10-10T00:37:13.1953385Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-10-10T00:37:13.1953526Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-10-10T00:37:13.1953654Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-10-10T00:37:13.1953785Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-10-10T00:37:13.1954105Z * [new branch] gh/pianpwk/1/base -> origin/gh/pianpwk/1/base 2025-10-10T00:37:13.1959100Z * [new branch] gh/pianpwk/1/head -> origin/gh/pianpwk/1/head 2025-10-10T00:37:13.1959345Z * [new branch] gh/pianpwk/1/orig -> origin/gh/pianpwk/1/orig 2025-10-10T00:37:13.1964637Z * [new branch] gh/pianpwk/2/base -> origin/gh/pianpwk/2/base 2025-10-10T00:37:13.1966664Z * [new branch] gh/pianpwk/2/head -> origin/gh/pianpwk/2/head 2025-10-10T00:37:13.1966976Z * [new branch] gh/pianpwk/2/orig -> origin/gh/pianpwk/2/orig 2025-10-10T00:37:13.1973489Z * [new branch] gh/pianpwk/3/base -> origin/gh/pianpwk/3/base 2025-10-10T00:37:13.1978505Z * [new branch] gh/pianpwk/3/head -> origin/gh/pianpwk/3/head 2025-10-10T00:37:13.1982955Z * [new branch] gh/pianpwk/3/orig -> origin/gh/pianpwk/3/orig 2025-10-10T00:37:13.1983143Z * [new branch] gh/pianpwk/4/base -> origin/gh/pianpwk/4/base 2025-10-10T00:37:13.1983308Z * [new branch] gh/pianpwk/4/head -> origin/gh/pianpwk/4/head 2025-10-10T00:37:13.1983446Z * [new branch] gh/pianpwk/4/orig -> origin/gh/pianpwk/4/orig 2025-10-10T00:37:13.1983583Z * [new branch] gh/pianpwk/5/base -> origin/gh/pianpwk/5/base 2025-10-10T00:37:13.1983731Z * [new branch] gh/pianpwk/5/head -> origin/gh/pianpwk/5/head 2025-10-10T00:37:13.1983887Z * [new branch] gh/pianpwk/5/orig -> origin/gh/pianpwk/5/orig 2025-10-10T00:37:13.1984030Z * [new branch] gh/pianpwk/6/base -> origin/gh/pianpwk/6/base 2025-10-10T00:37:13.1984168Z * [new branch] gh/pianpwk/6/head -> origin/gh/pianpwk/6/head 2025-10-10T00:37:13.1984312Z * [new branch] gh/pianpwk/6/orig -> origin/gh/pianpwk/6/orig 2025-10-10T00:37:13.1984611Z * [new branch] gh/pianpwk/7/base -> origin/gh/pianpwk/7/base 2025-10-10T00:37:13.1984749Z * [new branch] gh/pianpwk/7/head -> origin/gh/pianpwk/7/head 2025-10-10T00:37:13.1984916Z * [new branch] gh/pianpwk/7/orig -> origin/gh/pianpwk/7/orig 2025-10-10T00:37:13.1985058Z * [new branch] gh/pianpwk/8/base -> origin/gh/pianpwk/8/base 2025-10-10T00:37:13.1985208Z * [new branch] gh/pianpwk/8/head -> origin/gh/pianpwk/8/head 2025-10-10T00:37:13.1985352Z * [new branch] gh/pianpwk/8/orig -> origin/gh/pianpwk/8/orig 2025-10-10T00:37:13.1985541Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-10-10T00:37:13.1985685Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-10-10T00:37:13.1985821Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-10-10T00:37:13.1985965Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-10-10T00:37:13.1986093Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-10-10T00:37:13.1986229Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-10-10T00:37:13.1986599Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-10-10T00:37:13.1986741Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-10-10T00:37:13.1986926Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-10-10T00:37:13.1987056Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-10-10T00:37:13.1987196Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-10-10T00:37:13.1987322Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-10-10T00:37:13.1987462Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-10-10T00:37:13.1987588Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-10-10T00:37:13.1987716Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-10-10T00:37:13.1987852Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-10-10T00:37:13.1987978Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-10-10T00:37:13.1988115Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-10-10T00:37:13.1988291Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-10-10T00:37:13.1988463Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-10-10T00:37:13.1988622Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-10-10T00:37:13.1988783Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-10-10T00:37:13.1988945Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-10-10T00:37:13.1989100Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-10-10T00:37:13.1989267Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-10-10T00:37:13.1989425Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-10-10T00:37:13.1991255Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-10-10T00:37:13.1991496Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-10-10T00:37:13.1991673Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-10-10T00:37:13.1991882Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-10-10T00:37:13.1992101Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-10-10T00:37:13.1998561Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-10-10T00:37:13.2003386Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-10-10T00:37:13.2005428Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-10-10T00:37:13.2005622Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-10-10T00:37:13.2005878Z * [new branch] gh/rtimpe/15/base -> origin/gh/rtimpe/15/base 2025-10-10T00:37:13.2011358Z * [new branch] gh/rtimpe/15/head -> origin/gh/rtimpe/15/head 2025-10-10T00:37:13.2016298Z * [new branch] gh/rtimpe/15/orig -> origin/gh/rtimpe/15/orig 2025-10-10T00:37:13.2021005Z * [new branch] gh/rtimpe/16/base -> origin/gh/rtimpe/16/base 2025-10-10T00:37:13.2021332Z * [new branch] gh/rtimpe/16/head -> origin/gh/rtimpe/16/head 2025-10-10T00:37:13.2021516Z * [new branch] gh/rtimpe/16/orig -> origin/gh/rtimpe/16/orig 2025-10-10T00:37:13.2021679Z * [new branch] gh/rtimpe/17/base -> origin/gh/rtimpe/17/base 2025-10-10T00:37:13.2021835Z * [new branch] gh/rtimpe/17/head -> origin/gh/rtimpe/17/head 2025-10-10T00:37:13.2022114Z * [new branch] gh/rtimpe/17/orig -> origin/gh/rtimpe/17/orig 2025-10-10T00:37:13.2022255Z * [new branch] gh/rtimpe/18/base -> origin/gh/rtimpe/18/base 2025-10-10T00:37:13.2022506Z * [new branch] gh/rtimpe/18/head -> origin/gh/rtimpe/18/head 2025-10-10T00:37:13.2023148Z * [new branch] gh/rtimpe/18/orig -> origin/gh/rtimpe/18/orig 2025-10-10T00:37:13.2023353Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-10-10T00:37:13.2023602Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-10-10T00:37:13.2023759Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-10-10T00:37:13.2023911Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-10-10T00:37:13.2024071Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-10-10T00:37:13.2024221Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-10-10T00:37:13.2024415Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-10-10T00:37:13.2024583Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-10-10T00:37:13.2024762Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-10-10T00:37:13.2024919Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-10-10T00:37:13.2025085Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-10-10T00:37:13.2025241Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-10-10T00:37:13.2025399Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-10-10T00:37:13.2025564Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-10-10T00:37:13.2025722Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-10-10T00:37:13.2025878Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-10-10T00:37:13.2026031Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-10-10T00:37:13.2026249Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-10-10T00:37:13.2026669Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-10-10T00:37:13.2026836Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-10-10T00:37:13.2026995Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-10-10T00:37:13.2027142Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-10-10T00:37:13.2027309Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-10-10T00:37:13.2027458Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-10-10T00:37:13.2027657Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-10-10T00:37:13.2027829Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-10-10T00:37:13.2027996Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-10-10T00:37:13.2028145Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-10-10T00:37:13.2028286Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-10-10T00:37:13.2028430Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-10-10T00:37:13.2032442Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-10-10T00:37:13.2032771Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-10-10T00:37:13.2033010Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-10-10T00:37:13.2033275Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-10-10T00:37:13.2033504Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-10-10T00:37:13.2033672Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-10-10T00:37:13.2039866Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-10-10T00:37:13.2043921Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-10-10T00:37:13.2048127Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-10-10T00:37:13.2052993Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-10-10T00:37:13.2054913Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-10-10T00:37:13.2055178Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-10-10T00:37:13.2055388Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-10-10T00:37:13.2055615Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-10-10T00:37:13.2055830Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-10-10T00:37:13.2056039Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-10-10T00:37:13.2056298Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-10-10T00:37:13.2056501Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-10-10T00:37:13.2056662Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-10-10T00:37:13.2056866Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-10-10T00:37:13.2057102Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-10-10T00:37:13.2057340Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-10-10T00:37:13.2057662Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-10-10T00:37:13.2062563Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-10-10T00:37:13.2064797Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-10-10T00:37:13.2069039Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-10-10T00:37:13.2069189Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-10-10T00:37:13.2069364Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-10-10T00:37:13.2069505Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-10-10T00:37:13.2069648Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-10-10T00:37:13.2069786Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-10-10T00:37:13.2069929Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-10-10T00:37:13.2070071Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-10-10T00:37:13.2070206Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-10-10T00:37:13.2070348Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-10-10T00:37:13.2070483Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-10-10T00:37:13.2070754Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-10-10T00:37:13.2070905Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-10-10T00:37:13.2071042Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-10-10T00:37:13.2071189Z * [new branch] gh/seemethere/64/base -> origin/gh/seemethere/64/base 2025-10-10T00:37:13.2071325Z * [new branch] gh/seemethere/64/head -> origin/gh/seemethere/64/head 2025-10-10T00:37:13.2071476Z * [new branch] gh/seemethere/64/orig -> origin/gh/seemethere/64/orig 2025-10-10T00:37:13.2071611Z * [new branch] gh/seemethere/65/base -> origin/gh/seemethere/65/base 2025-10-10T00:37:13.2071744Z * [new branch] gh/seemethere/65/head -> origin/gh/seemethere/65/head 2025-10-10T00:37:13.2071892Z * [new branch] gh/seemethere/65/orig -> origin/gh/seemethere/65/orig 2025-10-10T00:37:13.2072027Z * [new branch] gh/seemethere/66/base -> origin/gh/seemethere/66/base 2025-10-10T00:37:13.2072170Z * [new branch] gh/seemethere/66/head -> origin/gh/seemethere/66/head 2025-10-10T00:37:13.2072305Z * [new branch] gh/seemethere/66/orig -> origin/gh/seemethere/66/orig 2025-10-10T00:37:13.2072450Z * [new branch] gh/seemethere/67/base -> origin/gh/seemethere/67/base 2025-10-10T00:37:13.2072585Z * [new branch] gh/seemethere/67/head -> origin/gh/seemethere/67/head 2025-10-10T00:37:13.2072719Z * [new branch] gh/seemethere/67/orig -> origin/gh/seemethere/67/orig 2025-10-10T00:37:13.2072863Z * [new branch] gh/seemethere/68/base -> origin/gh/seemethere/68/base 2025-10-10T00:37:13.2072999Z * [new branch] gh/seemethere/68/head -> origin/gh/seemethere/68/head 2025-10-10T00:37:13.2073147Z * [new branch] gh/seemethere/68/orig -> origin/gh/seemethere/68/orig 2025-10-10T00:37:13.2073287Z * [new branch] gh/seemethere/69/base -> origin/gh/seemethere/69/base 2025-10-10T00:37:13.2073430Z * [new branch] gh/seemethere/69/head -> origin/gh/seemethere/69/head 2025-10-10T00:37:13.2073564Z * [new branch] gh/seemethere/69/orig -> origin/gh/seemethere/69/orig 2025-10-10T00:37:13.2073843Z * [new branch] gh/seemethere/70/base -> origin/gh/seemethere/70/base 2025-10-10T00:37:13.2078929Z * [new branch] gh/seemethere/70/head -> origin/gh/seemethere/70/head 2025-10-10T00:37:13.2083272Z * [new branch] gh/seemethere/70/orig -> origin/gh/seemethere/70/orig 2025-10-10T00:37:13.2086933Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-10-10T00:37:13.2087118Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-10-10T00:37:13.2087470Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-10-10T00:37:13.2087639Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-10-10T00:37:13.2087789Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-10-10T00:37:13.2087935Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-10-10T00:37:13.2088102Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-10-10T00:37:13.2088239Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-10-10T00:37:13.2088374Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-10-10T00:37:13.2088557Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-10-10T00:37:13.2088821Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-10-10T00:37:13.2088965Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-10-10T00:37:13.2089128Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-10-10T00:37:13.2089270Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-10-10T00:37:13.2089407Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-10-10T00:37:13.2089541Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-10-10T00:37:13.2089684Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-10-10T00:37:13.2089819Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-10-10T00:37:13.2089959Z * [new branch] gh/shunting314/215/base -> origin/gh/shunting314/215/base 2025-10-10T00:37:13.2090097Z * [new branch] gh/shunting314/215/head -> origin/gh/shunting314/215/head 2025-10-10T00:37:13.2094134Z * [new branch] gh/shunting314/215/orig -> origin/gh/shunting314/215/orig 2025-10-10T00:37:13.2094302Z * [new branch] gh/shunting314/216/base -> origin/gh/shunting314/216/base 2025-10-10T00:37:13.2094456Z * [new branch] gh/shunting314/216/head -> origin/gh/shunting314/216/head 2025-10-10T00:37:13.2094607Z * [new branch] gh/shunting314/216/orig -> origin/gh/shunting314/216/orig 2025-10-10T00:37:13.2094744Z * [new branch] gh/shunting314/217/base -> origin/gh/shunting314/217/base 2025-10-10T00:37:13.2094885Z * [new branch] gh/shunting314/217/head -> origin/gh/shunting314/217/head 2025-10-10T00:37:13.2095018Z * [new branch] gh/shunting314/217/orig -> origin/gh/shunting314/217/orig 2025-10-10T00:37:13.2095465Z * [new branch] gh/shunting314/218/base -> origin/gh/shunting314/218/base 2025-10-10T00:37:13.2095607Z * [new branch] gh/shunting314/218/head -> origin/gh/shunting314/218/head 2025-10-10T00:37:13.2095883Z * [new branch] gh/shunting314/218/orig -> origin/gh/shunting314/218/orig 2025-10-10T00:37:13.2096035Z * [new branch] gh/shunting314/219/base -> origin/gh/shunting314/219/base 2025-10-10T00:37:13.2096321Z * [new branch] gh/shunting314/219/head -> origin/gh/shunting314/219/head 2025-10-10T00:37:13.2096470Z * [new branch] gh/shunting314/219/orig -> origin/gh/shunting314/219/orig 2025-10-10T00:37:13.2102932Z * [new branch] gh/shunting314/223/base -> origin/gh/shunting314/223/base 2025-10-10T00:37:13.2103204Z * [new branch] gh/shunting314/223/head -> origin/gh/shunting314/223/head 2025-10-10T00:37:13.2103366Z * [new branch] gh/shunting314/223/orig -> origin/gh/shunting314/223/orig 2025-10-10T00:37:13.2103590Z * [new branch] gh/shunting314/224/base -> origin/gh/shunting314/224/base 2025-10-10T00:37:13.2103842Z * [new branch] gh/shunting314/224/head -> origin/gh/shunting314/224/head 2025-10-10T00:37:13.2104095Z * [new branch] gh/shunting314/224/orig -> origin/gh/shunting314/224/orig 2025-10-10T00:37:13.2104324Z * [new branch] gh/shunting314/225/base -> origin/gh/shunting314/225/base 2025-10-10T00:37:13.2104546Z * [new branch] gh/shunting314/225/head -> origin/gh/shunting314/225/head 2025-10-10T00:37:13.2104726Z * [new branch] gh/shunting314/225/orig -> origin/gh/shunting314/225/orig 2025-10-10T00:37:13.2104878Z * [new branch] gh/shunting314/226/base -> origin/gh/shunting314/226/base 2025-10-10T00:37:13.2105399Z * [new branch] gh/shunting314/226/head -> origin/gh/shunting314/226/head 2025-10-10T00:37:13.2110542Z * [new branch] gh/shunting314/226/orig -> origin/gh/shunting314/226/orig 2025-10-10T00:37:13.2114683Z * [new branch] gh/shunting314/227/base -> origin/gh/shunting314/227/base 2025-10-10T00:37:13.2119018Z * [new branch] gh/shunting314/227/head -> origin/gh/shunting314/227/head 2025-10-10T00:37:13.2123896Z * [new branch] gh/shunting314/227/orig -> origin/gh/shunting314/227/orig 2025-10-10T00:37:13.2127531Z * [new branch] gh/shunting314/228/base -> origin/gh/shunting314/228/base 2025-10-10T00:37:13.2129352Z * [new branch] gh/shunting314/228/head -> origin/gh/shunting314/228/head 2025-10-10T00:37:13.2129534Z * [new branch] gh/shunting314/228/orig -> origin/gh/shunting314/228/orig 2025-10-10T00:37:13.2129685Z * [new branch] gh/shunting314/229/base -> origin/gh/shunting314/229/base 2025-10-10T00:37:13.2129838Z * [new branch] gh/shunting314/229/head -> origin/gh/shunting314/229/head 2025-10-10T00:37:13.2129993Z * [new branch] gh/shunting314/229/orig -> origin/gh/shunting314/229/orig 2025-10-10T00:37:13.2130154Z * [new branch] gh/shunting314/230/base -> origin/gh/shunting314/230/base 2025-10-10T00:37:13.2130294Z * [new branch] gh/shunting314/230/head -> origin/gh/shunting314/230/head 2025-10-10T00:37:13.2130435Z * [new branch] gh/shunting314/230/orig -> origin/gh/shunting314/230/orig 2025-10-10T00:37:13.2130593Z * [new branch] gh/shunting314/231/base -> origin/gh/shunting314/231/base 2025-10-10T00:37:13.2130734Z * [new branch] gh/shunting314/231/head -> origin/gh/shunting314/231/head 2025-10-10T00:37:13.2130882Z * [new branch] gh/shunting314/231/orig -> origin/gh/shunting314/231/orig 2025-10-10T00:37:13.2131023Z * [new branch] gh/shunting314/232/base -> origin/gh/shunting314/232/base 2025-10-10T00:37:13.2131171Z * [new branch] gh/shunting314/232/head -> origin/gh/shunting314/232/head 2025-10-10T00:37:13.2131312Z * [new branch] gh/shunting314/232/orig -> origin/gh/shunting314/232/orig 2025-10-10T00:37:13.2131632Z * [new branch] gh/shunting314/233/base -> origin/gh/shunting314/233/base 2025-10-10T00:37:13.2131789Z * [new branch] gh/shunting314/233/head -> origin/gh/shunting314/233/head 2025-10-10T00:37:13.2132153Z * [new branch] gh/shunting314/233/orig -> origin/gh/shunting314/233/orig 2025-10-10T00:37:13.2132303Z * [new branch] gh/shunting314/234/base -> origin/gh/shunting314/234/base 2025-10-10T00:37:13.2132456Z * [new branch] gh/shunting314/234/head -> origin/gh/shunting314/234/head 2025-10-10T00:37:13.2132594Z * [new branch] gh/shunting314/234/orig -> origin/gh/shunting314/234/orig 2025-10-10T00:37:13.2132740Z * [new branch] gh/shunting314/235/base -> origin/gh/shunting314/235/base 2025-10-10T00:37:13.2132882Z * [new branch] gh/shunting314/235/head -> origin/gh/shunting314/235/head 2025-10-10T00:37:13.2133027Z * [new branch] gh/shunting314/235/orig -> origin/gh/shunting314/235/orig 2025-10-10T00:37:13.2133176Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-10-10T00:37:13.2133323Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-10-10T00:37:13.2133460Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-10-10T00:37:13.2133596Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-10-10T00:37:13.2134841Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-10-10T00:37:13.2139870Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-10-10T00:37:13.2140195Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-10-10T00:37:13.2140570Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-10-10T00:37:13.2140848Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-10-10T00:37:13.2141052Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-10-10T00:37:13.2141261Z * [new branch] gh/slayton58/1/base -> origin/gh/slayton58/1/base 2025-10-10T00:37:13.2141868Z * [new branch] gh/slayton58/1/head -> origin/gh/slayton58/1/head 2025-10-10T00:37:13.2142058Z * [new branch] gh/slayton58/1/orig -> origin/gh/slayton58/1/orig 2025-10-10T00:37:13.2142216Z * [new branch] gh/slayton58/10/base -> origin/gh/slayton58/10/base 2025-10-10T00:37:13.2142361Z * [new branch] gh/slayton58/10/head -> origin/gh/slayton58/10/head 2025-10-10T00:37:13.2142525Z * [new branch] gh/slayton58/10/orig -> origin/gh/slayton58/10/orig 2025-10-10T00:37:13.2142661Z * [new branch] gh/slayton58/11/base -> origin/gh/slayton58/11/base 2025-10-10T00:37:13.2142802Z * [new branch] gh/slayton58/11/head -> origin/gh/slayton58/11/head 2025-10-10T00:37:13.2142937Z * [new branch] gh/slayton58/11/orig -> origin/gh/slayton58/11/orig 2025-10-10T00:37:13.2143104Z * [new branch] gh/slayton58/12/base -> origin/gh/slayton58/12/base 2025-10-10T00:37:13.2143241Z * [new branch] gh/slayton58/12/head -> origin/gh/slayton58/12/head 2025-10-10T00:37:13.2143561Z * [new branch] gh/slayton58/12/orig -> origin/gh/slayton58/12/orig 2025-10-10T00:37:13.2143905Z * [new branch] gh/slayton58/13/base -> origin/gh/slayton58/13/base 2025-10-10T00:37:13.2145072Z * [new branch] gh/slayton58/13/head -> origin/gh/slayton58/13/head 2025-10-10T00:37:13.2145691Z * [new branch] gh/slayton58/13/orig -> origin/gh/slayton58/13/orig 2025-10-10T00:37:13.2146946Z * [new branch] gh/slayton58/14/base -> origin/gh/slayton58/14/base 2025-10-10T00:37:13.2150341Z * [new branch] gh/slayton58/14/head -> origin/gh/slayton58/14/head 2025-10-10T00:37:13.2156684Z * [new branch] gh/slayton58/14/orig -> origin/gh/slayton58/14/orig 2025-10-10T00:37:13.2162771Z * [new branch] gh/slayton58/15/base -> origin/gh/slayton58/15/base 2025-10-10T00:37:13.2166215Z * [new branch] gh/slayton58/15/head -> origin/gh/slayton58/15/head 2025-10-10T00:37:13.2171161Z * [new branch] gh/slayton58/15/orig -> origin/gh/slayton58/15/orig 2025-10-10T00:37:13.2173070Z * [new branch] gh/slayton58/16/base -> origin/gh/slayton58/16/base 2025-10-10T00:37:13.2173237Z * [new branch] gh/slayton58/16/head -> origin/gh/slayton58/16/head 2025-10-10T00:37:13.2173401Z * [new branch] gh/slayton58/16/orig -> origin/gh/slayton58/16/orig 2025-10-10T00:37:13.2173539Z * [new branch] gh/slayton58/17/base -> origin/gh/slayton58/17/base 2025-10-10T00:37:13.2173682Z * [new branch] gh/slayton58/17/head -> origin/gh/slayton58/17/head 2025-10-10T00:37:13.2173833Z * [new branch] gh/slayton58/17/orig -> origin/gh/slayton58/17/orig 2025-10-10T00:37:13.2173990Z * [new branch] gh/slayton58/18/base -> origin/gh/slayton58/18/base 2025-10-10T00:37:13.2174149Z * [new branch] gh/slayton58/18/head -> origin/gh/slayton58/18/head 2025-10-10T00:37:13.2174299Z * [new branch] gh/slayton58/19/base -> origin/gh/slayton58/19/base 2025-10-10T00:37:13.2174453Z * [new branch] gh/slayton58/19/head -> origin/gh/slayton58/19/head 2025-10-10T00:37:13.2174604Z * [new branch] gh/slayton58/19/orig -> origin/gh/slayton58/19/orig 2025-10-10T00:37:13.2174935Z * [new branch] gh/slayton58/2/base -> origin/gh/slayton58/2/base 2025-10-10T00:37:13.2175102Z * [new branch] gh/slayton58/2/head -> origin/gh/slayton58/2/head 2025-10-10T00:37:13.2175253Z * [new branch] gh/slayton58/2/orig -> origin/gh/slayton58/2/orig 2025-10-10T00:37:13.2175433Z * [new branch] gh/slayton58/20/base -> origin/gh/slayton58/20/base 2025-10-10T00:37:13.2175568Z * [new branch] gh/slayton58/20/head -> origin/gh/slayton58/20/head 2025-10-10T00:37:13.2175705Z * [new branch] gh/slayton58/20/orig -> origin/gh/slayton58/20/orig 2025-10-10T00:37:13.2175836Z * [new branch] gh/slayton58/21/base -> origin/gh/slayton58/21/base 2025-10-10T00:37:13.2175968Z * [new branch] gh/slayton58/21/head -> origin/gh/slayton58/21/head 2025-10-10T00:37:13.2176110Z * [new branch] gh/slayton58/21/orig -> origin/gh/slayton58/21/orig 2025-10-10T00:37:13.2176243Z * [new branch] gh/slayton58/22/base -> origin/gh/slayton58/22/base 2025-10-10T00:37:13.2176377Z * [new branch] gh/slayton58/22/head -> origin/gh/slayton58/22/head 2025-10-10T00:37:13.2176509Z * [new branch] gh/slayton58/22/orig -> origin/gh/slayton58/22/orig 2025-10-10T00:37:13.2176650Z * [new branch] gh/slayton58/23/base -> origin/gh/slayton58/23/base 2025-10-10T00:37:13.2176786Z * [new branch] gh/slayton58/23/head -> origin/gh/slayton58/23/head 2025-10-10T00:37:13.2178924Z * [new branch] gh/slayton58/23/orig -> origin/gh/slayton58/23/orig 2025-10-10T00:37:13.2179230Z * [new branch] gh/slayton58/24/base -> origin/gh/slayton58/24/base 2025-10-10T00:37:13.2179389Z * [new branch] gh/slayton58/24/head -> origin/gh/slayton58/24/head 2025-10-10T00:37:13.2179628Z * [new branch] gh/slayton58/24/orig -> origin/gh/slayton58/24/orig 2025-10-10T00:37:13.2179781Z * [new branch] gh/slayton58/25/base -> origin/gh/slayton58/25/base 2025-10-10T00:37:13.2180020Z * [new branch] gh/slayton58/25/head -> origin/gh/slayton58/25/head 2025-10-10T00:37:13.2180169Z * [new branch] gh/slayton58/25/orig -> origin/gh/slayton58/25/orig 2025-10-10T00:37:13.2180575Z * [new branch] gh/slayton58/26/base -> origin/gh/slayton58/26/base 2025-10-10T00:37:13.2180742Z * [new branch] gh/slayton58/26/head -> origin/gh/slayton58/26/head 2025-10-10T00:37:13.2180955Z * [new branch] gh/slayton58/26/orig -> origin/gh/slayton58/26/orig 2025-10-10T00:37:13.2181245Z * [new branch] gh/slayton58/3/base -> origin/gh/slayton58/3/base 2025-10-10T00:37:13.2181917Z * [new branch] gh/slayton58/3/head -> origin/gh/slayton58/3/head 2025-10-10T00:37:13.2182150Z * [new branch] gh/slayton58/3/orig -> origin/gh/slayton58/3/orig 2025-10-10T00:37:13.2182300Z * [new branch] gh/slayton58/4/base -> origin/gh/slayton58/4/base 2025-10-10T00:37:13.2182462Z * [new branch] gh/slayton58/4/head -> origin/gh/slayton58/4/head 2025-10-10T00:37:13.2182614Z * [new branch] gh/slayton58/4/orig -> origin/gh/slayton58/4/orig 2025-10-10T00:37:13.2182791Z * [new branch] gh/slayton58/5/base -> origin/gh/slayton58/5/base 2025-10-10T00:37:13.2182938Z * [new branch] gh/slayton58/5/head -> origin/gh/slayton58/5/head 2025-10-10T00:37:13.2183085Z * [new branch] gh/slayton58/5/orig -> origin/gh/slayton58/5/orig 2025-10-10T00:37:13.2183415Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-10-10T00:37:13.2184077Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-10-10T00:37:13.2184434Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-10-10T00:37:13.2184690Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-10-10T00:37:13.2185853Z * [new branch] gh/slayton58/8/base -> origin/gh/slayton58/8/base 2025-10-10T00:37:13.2192644Z * [new branch] gh/slayton58/8/head -> origin/gh/slayton58/8/head 2025-10-10T00:37:13.2198833Z * [new branch] gh/slayton58/8/orig -> origin/gh/slayton58/8/orig 2025-10-10T00:37:13.2200737Z * [new branch] gh/slayton58/9/base -> origin/gh/slayton58/9/base 2025-10-10T00:37:13.2200990Z * [new branch] gh/slayton58/9/head -> origin/gh/slayton58/9/head 2025-10-10T00:37:13.2206577Z * [new branch] gh/slayton58/9/orig -> origin/gh/slayton58/9/orig 2025-10-10T00:37:13.2212738Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-10-10T00:37:13.2217537Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-10-10T00:37:13.2219537Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-10-10T00:37:13.2219873Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-10-10T00:37:13.2220107Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-10-10T00:37:13.2220341Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-10-10T00:37:13.2220493Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-10-10T00:37:13.2220713Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-10-10T00:37:13.2221377Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-10-10T00:37:13.2221578Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-10-10T00:37:13.2221730Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-10-10T00:37:13.2221869Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-10-10T00:37:13.2222017Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-10-10T00:37:13.2222298Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-10-10T00:37:13.2222443Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-10-10T00:37:13.2222586Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-10-10T00:37:13.2222725Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-10-10T00:37:13.2222870Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-10-10T00:37:13.2223011Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-10-10T00:37:13.2223157Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-10-10T00:37:13.2223297Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-10-10T00:37:13.2223439Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-10-10T00:37:13.2223587Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-10-10T00:37:13.2223739Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-10-10T00:37:13.2223894Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-10-10T00:37:13.2224050Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-10-10T00:37:13.2224210Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-10-10T00:37:13.2224399Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-10-10T00:37:13.2224565Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-10-10T00:37:13.2224724Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-10-10T00:37:13.2224875Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-10-10T00:37:13.2225028Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-10-10T00:37:13.2225180Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-10-10T00:37:13.2225333Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-10-10T00:37:13.2225493Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-10-10T00:37:13.2225655Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-10-10T00:37:13.2225806Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-10-10T00:37:13.2225958Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-10-10T00:37:13.2226116Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-10-10T00:37:13.2226543Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-10-10T00:37:13.2226716Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-10-10T00:37:13.2226879Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-10-10T00:37:13.2227044Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-10-10T00:37:13.2227200Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-10-10T00:37:13.2227363Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-10-10T00:37:13.2227515Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-10-10T00:37:13.2227667Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-10-10T00:37:13.2227819Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-10-10T00:37:13.2228050Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-10-10T00:37:13.2235680Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-10-10T00:37:13.2239077Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-10-10T00:37:13.2239357Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-10-10T00:37:13.2244307Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-10-10T00:37:13.2247821Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-10-10T00:37:13.2248109Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-10-10T00:37:13.2253942Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-10-10T00:37:13.2258414Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-10-10T00:37:13.2259585Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-10-10T00:37:13.2259730Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-10-10T00:37:13.2259902Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-10-10T00:37:13.2260042Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-10-10T00:37:13.2260380Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-10-10T00:37:13.2260523Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-10-10T00:37:13.2260660Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-10-10T00:37:13.2260805Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-10-10T00:37:13.2260947Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-10-10T00:37:13.2261091Z * [new branch] gh/soulitzer/381/base -> origin/gh/soulitzer/381/base 2025-10-10T00:37:13.2261228Z * [new branch] gh/soulitzer/381/head -> origin/gh/soulitzer/381/head 2025-10-10T00:37:13.2261370Z * [new branch] gh/soulitzer/381/orig -> origin/gh/soulitzer/381/orig 2025-10-10T00:37:13.2261505Z * [new branch] gh/soulitzer/382/base -> origin/gh/soulitzer/382/base 2025-10-10T00:37:13.2261645Z * [new branch] gh/soulitzer/382/head -> origin/gh/soulitzer/382/head 2025-10-10T00:37:13.2261793Z * [new branch] gh/soulitzer/382/orig -> origin/gh/soulitzer/382/orig 2025-10-10T00:37:13.2261928Z * [new branch] gh/soulitzer/383/base -> origin/gh/soulitzer/383/base 2025-10-10T00:37:13.2262075Z * [new branch] gh/soulitzer/383/head -> origin/gh/soulitzer/383/head 2025-10-10T00:37:13.2262220Z * [new branch] gh/soulitzer/383/orig -> origin/gh/soulitzer/383/orig 2025-10-10T00:37:13.2262368Z * [new branch] gh/soulitzer/384/base -> origin/gh/soulitzer/384/base 2025-10-10T00:37:13.2262511Z * [new branch] gh/soulitzer/384/head -> origin/gh/soulitzer/384/head 2025-10-10T00:37:13.2262653Z * [new branch] gh/soulitzer/384/orig -> origin/gh/soulitzer/384/orig 2025-10-10T00:37:13.2262811Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-10-10T00:37:13.2262969Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-10-10T00:37:13.2263119Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-10-10T00:37:13.2263261Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-10-10T00:37:13.2263400Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-10-10T00:37:13.2263603Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-10-10T00:37:13.2263742Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-10-10T00:37:13.2263888Z * [new branch] gh/swolchok/809/base -> origin/gh/swolchok/809/base 2025-10-10T00:37:13.2264027Z * [new branch] gh/swolchok/809/head -> origin/gh/swolchok/809/head 2025-10-10T00:37:13.2264177Z * [new branch] gh/swolchok/809/orig -> origin/gh/swolchok/809/orig 2025-10-10T00:37:13.2264318Z * [new branch] gh/swolchok/815/base -> origin/gh/swolchok/815/base 2025-10-10T00:37:13.2264458Z * [new branch] gh/swolchok/815/head -> origin/gh/swolchok/815/head 2025-10-10T00:37:13.2264619Z * [new branch] gh/swolchok/815/orig -> origin/gh/swolchok/815/orig 2025-10-10T00:37:13.2264765Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-10-10T00:37:13.2264916Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-10-10T00:37:13.2265058Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-10-10T00:37:13.2265208Z * [new branch] gh/swolchok/821/base -> origin/gh/swolchok/821/base 2025-10-10T00:37:13.2265353Z * [new branch] gh/swolchok/821/head -> origin/gh/swolchok/821/head 2025-10-10T00:37:13.2265625Z * [new branch] gh/swolchok/821/orig -> origin/gh/swolchok/821/orig 2025-10-10T00:37:13.2265787Z * [new branch] gh/swolchok/823/base -> origin/gh/swolchok/823/base 2025-10-10T00:37:13.2265931Z * [new branch] gh/swolchok/823/head -> origin/gh/swolchok/823/head 2025-10-10T00:37:13.2266086Z * [new branch] gh/swolchok/823/orig -> origin/gh/swolchok/823/orig 2025-10-10T00:37:13.2266241Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-10-10T00:37:13.2267782Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-10-10T00:37:13.2276458Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-10-10T00:37:13.2278561Z * [new branch] gh/swolchok/826/base -> origin/gh/swolchok/826/base 2025-10-10T00:37:13.2278850Z * [new branch] gh/swolchok/826/head -> origin/gh/swolchok/826/head 2025-10-10T00:37:13.2282883Z * [new branch] gh/swolchok/826/orig -> origin/gh/swolchok/826/orig 2025-10-10T00:37:13.2283147Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-10-10T00:37:13.2287952Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-10-10T00:37:13.2293127Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-10-10T00:37:13.2295337Z * [new branch] gh/swolchok/830/base -> origin/gh/swolchok/830/base 2025-10-10T00:37:13.2295538Z * [new branch] gh/swolchok/830/head -> origin/gh/swolchok/830/head 2025-10-10T00:37:13.2295686Z * [new branch] gh/swolchok/830/orig -> origin/gh/swolchok/830/orig 2025-10-10T00:37:13.2295838Z * [new branch] gh/swolchok/831/base -> origin/gh/swolchok/831/base 2025-10-10T00:37:13.2295982Z * [new branch] gh/swolchok/831/head -> origin/gh/swolchok/831/head 2025-10-10T00:37:13.2296158Z * [new branch] gh/swolchok/831/orig -> origin/gh/swolchok/831/orig 2025-10-10T00:37:13.2296290Z * [new branch] gh/swolchok/832/base -> origin/gh/swolchok/832/base 2025-10-10T00:37:13.2296422Z * [new branch] gh/swolchok/832/head -> origin/gh/swolchok/832/head 2025-10-10T00:37:13.2296566Z * [new branch] gh/swolchok/832/orig -> origin/gh/swolchok/832/orig 2025-10-10T00:37:13.2296860Z * [new branch] gh/swolchok/833/base -> origin/gh/swolchok/833/base 2025-10-10T00:37:13.2297005Z * [new branch] gh/swolchok/833/head -> origin/gh/swolchok/833/head 2025-10-10T00:37:13.2297137Z * [new branch] gh/swolchok/833/orig -> origin/gh/swolchok/833/orig 2025-10-10T00:37:13.2297280Z * [new branch] gh/swolchok/834/base -> origin/gh/swolchok/834/base 2025-10-10T00:37:13.2297426Z * [new branch] gh/swolchok/834/head -> origin/gh/swolchok/834/head 2025-10-10T00:37:13.2297561Z * [new branch] gh/swolchok/834/orig -> origin/gh/swolchok/834/orig 2025-10-10T00:37:13.2297703Z * [new branch] gh/swolchok/835/base -> origin/gh/swolchok/835/base 2025-10-10T00:37:13.2297832Z * [new branch] gh/swolchok/835/head -> origin/gh/swolchok/835/head 2025-10-10T00:37:13.2297973Z * [new branch] gh/swolchok/835/orig -> origin/gh/swolchok/835/orig 2025-10-10T00:37:13.2298106Z * [new branch] gh/swolchok/836/base -> origin/gh/swolchok/836/base 2025-10-10T00:37:13.2298236Z * [new branch] gh/swolchok/836/head -> origin/gh/swolchok/836/head 2025-10-10T00:37:13.2298375Z * [new branch] gh/swolchok/836/orig -> origin/gh/swolchok/836/orig 2025-10-10T00:37:13.2298511Z * [new branch] gh/swolchok/837/base -> origin/gh/swolchok/837/base 2025-10-10T00:37:13.2298684Z * [new branch] gh/swolchok/837/head -> origin/gh/swolchok/837/head 2025-10-10T00:37:13.2298812Z * [new branch] gh/swolchok/837/orig -> origin/gh/swolchok/837/orig 2025-10-10T00:37:13.2298947Z * [new branch] gh/swolchok/838/base -> origin/gh/swolchok/838/base 2025-10-10T00:37:13.2299075Z * [new branch] gh/swolchok/838/head -> origin/gh/swolchok/838/head 2025-10-10T00:37:13.2299206Z * [new branch] gh/swolchok/838/orig -> origin/gh/swolchok/838/orig 2025-10-10T00:37:13.2299343Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-10-10T00:37:13.2299472Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-10-10T00:37:13.2299608Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-10-10T00:37:13.2299735Z * [new branch] gh/swolchok/840/base -> origin/gh/swolchok/840/base 2025-10-10T00:37:13.2299873Z * [new branch] gh/swolchok/840/head -> origin/gh/swolchok/840/head 2025-10-10T00:37:13.2300003Z * [new branch] gh/swolchok/840/orig -> origin/gh/swolchok/840/orig 2025-10-10T00:37:13.2302901Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-10-10T00:37:13.2303068Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-10-10T00:37:13.2303209Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-10-10T00:37:13.2303496Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-10-10T00:37:13.2303652Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-10-10T00:37:13.2303882Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-10-10T00:37:13.2304042Z * [new branch] gh/swolchok/843/base -> origin/gh/swolchok/843/base 2025-10-10T00:37:13.2304284Z * [new branch] gh/swolchok/843/head -> origin/gh/swolchok/843/head 2025-10-10T00:37:13.2304964Z * [new branch] gh/swolchok/843/orig -> origin/gh/swolchok/843/orig 2025-10-10T00:37:13.2305165Z * [new branch] gh/swolchok/844/base -> origin/gh/swolchok/844/base 2025-10-10T00:37:13.2305587Z * [new branch] gh/swolchok/844/head -> origin/gh/swolchok/844/head 2025-10-10T00:37:13.2305814Z * [new branch] gh/swolchok/844/orig -> origin/gh/swolchok/844/orig 2025-10-10T00:37:13.2305961Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-10-10T00:37:13.2306339Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-10-10T00:37:13.2307704Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-10-10T00:37:13.2311963Z * [new branch] gh/swolchok/846/base -> origin/gh/swolchok/846/base 2025-10-10T00:37:13.2312263Z * [new branch] gh/swolchok/846/head -> origin/gh/swolchok/846/head 2025-10-10T00:37:13.2318528Z * [new branch] gh/swolchok/846/orig -> origin/gh/swolchok/846/orig 2025-10-10T00:37:13.2320455Z * [new branch] gh/swolchok/847/base -> origin/gh/swolchok/847/base 2025-10-10T00:37:13.2320757Z * [new branch] gh/swolchok/847/head -> origin/gh/swolchok/847/head 2025-10-10T00:37:13.2324376Z * [new branch] gh/swolchok/847/orig -> origin/gh/swolchok/847/orig 2025-10-10T00:37:13.2324672Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-10-10T00:37:13.2330366Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-10-10T00:37:13.2336087Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-10-10T00:37:13.2340130Z * [new branch] gh/swolchok/849/base -> origin/gh/swolchok/849/base 2025-10-10T00:37:13.2340500Z * [new branch] gh/swolchok/849/head -> origin/gh/swolchok/849/head 2025-10-10T00:37:13.2340702Z * [new branch] gh/swolchok/849/orig -> origin/gh/swolchok/849/orig 2025-10-10T00:37:13.2340948Z * [new branch] gh/swolchok/850/base -> origin/gh/swolchok/850/base 2025-10-10T00:37:13.2341156Z * [new branch] gh/swolchok/850/head -> origin/gh/swolchok/850/head 2025-10-10T00:37:13.2341809Z * [new branch] gh/swolchok/850/orig -> origin/gh/swolchok/850/orig 2025-10-10T00:37:13.2342734Z * [new branch] gh/swolchok/851/base -> origin/gh/swolchok/851/base 2025-10-10T00:37:13.2342888Z * [new branch] gh/swolchok/851/head -> origin/gh/swolchok/851/head 2025-10-10T00:37:13.2343061Z * [new branch] gh/swolchok/851/orig -> origin/gh/swolchok/851/orig 2025-10-10T00:37:13.2343226Z * [new branch] gh/swolchok/852/base -> origin/gh/swolchok/852/base 2025-10-10T00:37:13.2343370Z * [new branch] gh/swolchok/852/head -> origin/gh/swolchok/852/head 2025-10-10T00:37:13.2343540Z * [new branch] gh/swolchok/852/orig -> origin/gh/swolchok/852/orig 2025-10-10T00:37:13.2343704Z * [new branch] gh/syed-ahmed/5/base -> origin/gh/syed-ahmed/5/base 2025-10-10T00:37:13.2343868Z * [new branch] gh/syed-ahmed/5/head -> origin/gh/syed-ahmed/5/head 2025-10-10T00:37:13.2344010Z * [new branch] gh/syed-ahmed/5/orig -> origin/gh/syed-ahmed/5/orig 2025-10-10T00:37:13.2344160Z * [new branch] gh/syed-ahmed/6/base -> origin/gh/syed-ahmed/6/base 2025-10-10T00:37:13.2344303Z * [new branch] gh/syed-ahmed/6/head -> origin/gh/syed-ahmed/6/head 2025-10-10T00:37:13.2344456Z * [new branch] gh/syed-ahmed/6/orig -> origin/gh/syed-ahmed/6/orig 2025-10-10T00:37:13.2344606Z * [new branch] gh/syed-ahmed/7/base -> origin/gh/syed-ahmed/7/base 2025-10-10T00:37:13.2344744Z * [new branch] gh/syed-ahmed/7/head -> origin/gh/syed-ahmed/7/head 2025-10-10T00:37:13.2344893Z * [new branch] gh/syed-ahmed/7/orig -> origin/gh/syed-ahmed/7/orig 2025-10-10T00:37:13.2345041Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-10-10T00:37:13.2345378Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-10-10T00:37:13.2345518Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-10-10T00:37:13.2345661Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-10-10T00:37:13.2345808Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-10-10T00:37:13.2345944Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-10-10T00:37:13.2346094Z * [new branch] gh/tianyu-l/5/base -> origin/gh/tianyu-l/5/base 2025-10-10T00:37:13.2346231Z * [new branch] gh/tianyu-l/5/orig -> origin/gh/tianyu-l/5/orig 2025-10-10T00:37:13.2346547Z * [new branch] gh/tianyu-l/6/base -> origin/gh/tianyu-l/6/base 2025-10-10T00:37:13.2346694Z * [new branch] gh/tianyu-l/6/head -> origin/gh/tianyu-l/6/head 2025-10-10T00:37:13.2346842Z * [new branch] gh/tianyu-l/6/orig -> origin/gh/tianyu-l/6/orig 2025-10-10T00:37:13.2346988Z * [new branch] gh/tianyu-l/7/base -> origin/gh/tianyu-l/7/base 2025-10-10T00:37:13.2347123Z * [new branch] gh/tianyu-l/7/orig -> origin/gh/tianyu-l/7/orig 2025-10-10T00:37:13.2347312Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-10-10T00:37:13.2352455Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-10-10T00:37:13.2354646Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-10-10T00:37:13.2354962Z * [new branch] gh/tugsbayasgalan/11/base -> origin/gh/tugsbayasgalan/11/base 2025-10-10T00:37:13.2355439Z * [new branch] gh/tugsbayasgalan/11/head -> origin/gh/tugsbayasgalan/11/head 2025-10-10T00:37:13.2355642Z * [new branch] gh/tugsbayasgalan/11/orig -> origin/gh/tugsbayasgalan/11/orig 2025-10-10T00:37:13.2355823Z * [new branch] gh/tugsbayasgalan/12/base -> origin/gh/tugsbayasgalan/12/base 2025-10-10T00:37:13.2355979Z * [new branch] gh/tugsbayasgalan/12/head -> origin/gh/tugsbayasgalan/12/head 2025-10-10T00:37:13.2356157Z * [new branch] gh/tugsbayasgalan/12/orig -> origin/gh/tugsbayasgalan/12/orig 2025-10-10T00:37:13.2356333Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-10-10T00:37:13.2356505Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-10-10T00:37:13.2356672Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-10-10T00:37:13.2356838Z * [new branch] gh/tugsbayasgalan/14/base -> origin/gh/tugsbayasgalan/14/base 2025-10-10T00:37:13.2358675Z * [new branch] gh/tugsbayasgalan/14/head -> origin/gh/tugsbayasgalan/14/head 2025-10-10T00:37:13.2359275Z * [new branch] gh/tugsbayasgalan/14/orig -> origin/gh/tugsbayasgalan/14/orig 2025-10-10T00:37:13.2359456Z * [new branch] gh/tugsbayasgalan/15/base -> origin/gh/tugsbayasgalan/15/base 2025-10-10T00:37:13.2359811Z * [new branch] gh/tugsbayasgalan/15/head -> origin/gh/tugsbayasgalan/15/head 2025-10-10T00:37:13.2359969Z * [new branch] gh/tugsbayasgalan/15/orig -> origin/gh/tugsbayasgalan/15/orig 2025-10-10T00:37:13.2364072Z * [new branch] gh/tugsbayasgalan/16/base -> origin/gh/tugsbayasgalan/16/base 2025-10-10T00:37:13.2364309Z * [new branch] gh/tugsbayasgalan/16/head -> origin/gh/tugsbayasgalan/16/head 2025-10-10T00:37:13.2364468Z * [new branch] gh/tugsbayasgalan/16/orig -> origin/gh/tugsbayasgalan/16/orig 2025-10-10T00:37:13.2364626Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-10-10T00:37:13.2364922Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-10-10T00:37:13.2365084Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-10-10T00:37:13.2365975Z * [new branch] gh/tugsbayasgalan/18/base -> origin/gh/tugsbayasgalan/18/base 2025-10-10T00:37:13.2370596Z * [new branch] gh/tugsbayasgalan/18/head -> origin/gh/tugsbayasgalan/18/head 2025-10-10T00:37:13.2370847Z * [new branch] gh/tugsbayasgalan/18/orig -> origin/gh/tugsbayasgalan/18/orig 2025-10-10T00:37:13.2371102Z * [new branch] gh/tugsbayasgalan/19/base -> origin/gh/tugsbayasgalan/19/base 2025-10-10T00:37:13.2371259Z * [new branch] gh/tugsbayasgalan/19/head -> origin/gh/tugsbayasgalan/19/head 2025-10-10T00:37:13.2371412Z * [new branch] gh/tugsbayasgalan/19/orig -> origin/gh/tugsbayasgalan/19/orig 2025-10-10T00:37:13.2379824Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-10-10T00:37:13.2383767Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-10-10T00:37:13.2383972Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-10-10T00:37:13.2384160Z * [new branch] gh/tugsbayasgalan/20/base -> origin/gh/tugsbayasgalan/20/base 2025-10-10T00:37:13.2384339Z * [new branch] gh/tugsbayasgalan/20/head -> origin/gh/tugsbayasgalan/20/head 2025-10-10T00:37:13.2384676Z * [new branch] gh/tugsbayasgalan/20/orig -> origin/gh/tugsbayasgalan/20/orig 2025-10-10T00:37:13.2384877Z * [new branch] gh/tugsbayasgalan/21/base -> origin/gh/tugsbayasgalan/21/base 2025-10-10T00:37:13.2385047Z * [new branch] gh/tugsbayasgalan/21/head -> origin/gh/tugsbayasgalan/21/head 2025-10-10T00:37:13.2385216Z * [new branch] gh/tugsbayasgalan/21/orig -> origin/gh/tugsbayasgalan/21/orig 2025-10-10T00:37:13.2385411Z * [new branch] gh/tugsbayasgalan/22/base -> origin/gh/tugsbayasgalan/22/base 2025-10-10T00:37:13.2385577Z * [new branch] gh/tugsbayasgalan/22/head -> origin/gh/tugsbayasgalan/22/head 2025-10-10T00:37:13.2385744Z * [new branch] gh/tugsbayasgalan/22/orig -> origin/gh/tugsbayasgalan/22/orig 2025-10-10T00:37:13.2385909Z * [new branch] gh/tugsbayasgalan/23/base -> origin/gh/tugsbayasgalan/23/base 2025-10-10T00:37:13.2386087Z * [new branch] gh/tugsbayasgalan/23/head -> origin/gh/tugsbayasgalan/23/head 2025-10-10T00:37:13.2386246Z * [new branch] gh/tugsbayasgalan/23/orig -> origin/gh/tugsbayasgalan/23/orig 2025-10-10T00:37:13.2386600Z * [new branch] gh/tugsbayasgalan/24/base -> origin/gh/tugsbayasgalan/24/base 2025-10-10T00:37:13.2386775Z * [new branch] gh/tugsbayasgalan/24/head -> origin/gh/tugsbayasgalan/24/head 2025-10-10T00:37:13.2386944Z * [new branch] gh/tugsbayasgalan/24/orig -> origin/gh/tugsbayasgalan/24/orig 2025-10-10T00:37:13.2387123Z * [new branch] gh/tugsbayasgalan/25/base -> origin/gh/tugsbayasgalan/25/base 2025-10-10T00:37:13.2387295Z * [new branch] gh/tugsbayasgalan/25/head -> origin/gh/tugsbayasgalan/25/head 2025-10-10T00:37:13.2387462Z * [new branch] gh/tugsbayasgalan/25/orig -> origin/gh/tugsbayasgalan/25/orig 2025-10-10T00:37:13.2387622Z * [new branch] gh/tugsbayasgalan/26/base -> origin/gh/tugsbayasgalan/26/base 2025-10-10T00:37:13.2387777Z * [new branch] gh/tugsbayasgalan/26/head -> origin/gh/tugsbayasgalan/26/head 2025-10-10T00:37:13.2387933Z * [new branch] gh/tugsbayasgalan/26/orig -> origin/gh/tugsbayasgalan/26/orig 2025-10-10T00:37:13.2392605Z * [new branch] gh/tugsbayasgalan/27/base -> origin/gh/tugsbayasgalan/27/base 2025-10-10T00:37:13.2397555Z * [new branch] gh/tugsbayasgalan/27/head -> origin/gh/tugsbayasgalan/27/head 2025-10-10T00:37:13.2402683Z * [new branch] gh/tugsbayasgalan/27/orig -> origin/gh/tugsbayasgalan/27/orig 2025-10-10T00:37:13.2407731Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-10-10T00:37:13.2408079Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-10-10T00:37:13.2408343Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-10-10T00:37:13.2408606Z * [new branch] gh/tugsbayasgalan/29/base -> origin/gh/tugsbayasgalan/29/base 2025-10-10T00:37:13.2408777Z * [new branch] gh/tugsbayasgalan/29/head -> origin/gh/tugsbayasgalan/29/head 2025-10-10T00:37:13.2408934Z * [new branch] gh/tugsbayasgalan/29/orig -> origin/gh/tugsbayasgalan/29/orig 2025-10-10T00:37:13.2409220Z * [new branch] gh/tugsbayasgalan/3/base -> origin/gh/tugsbayasgalan/3/base 2025-10-10T00:37:13.2409960Z * [new branch] gh/tugsbayasgalan/3/head -> origin/gh/tugsbayasgalan/3/head 2025-10-10T00:37:13.2410189Z * [new branch] gh/tugsbayasgalan/3/orig -> origin/gh/tugsbayasgalan/3/orig 2025-10-10T00:37:13.2410395Z * [new branch] gh/tugsbayasgalan/30/base -> origin/gh/tugsbayasgalan/30/base 2025-10-10T00:37:13.2410572Z * [new branch] gh/tugsbayasgalan/30/head -> origin/gh/tugsbayasgalan/30/head 2025-10-10T00:37:13.2410747Z * [new branch] gh/tugsbayasgalan/30/orig -> origin/gh/tugsbayasgalan/30/orig 2025-10-10T00:37:13.2411071Z * [new branch] gh/tugsbayasgalan/31/base -> origin/gh/tugsbayasgalan/31/base 2025-10-10T00:37:13.2411249Z * [new branch] gh/tugsbayasgalan/31/head -> origin/gh/tugsbayasgalan/31/head 2025-10-10T00:37:13.2411428Z * [new branch] gh/tugsbayasgalan/31/orig -> origin/gh/tugsbayasgalan/31/orig 2025-10-10T00:37:13.2411608Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-10-10T00:37:13.2411780Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-10-10T00:37:13.2411953Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-10-10T00:37:13.2412135Z * [new branch] gh/tugsbayasgalan/33/base -> origin/gh/tugsbayasgalan/33/base 2025-10-10T00:37:13.2412305Z * [new branch] gh/tugsbayasgalan/33/head -> origin/gh/tugsbayasgalan/33/head 2025-10-10T00:37:13.2412480Z * [new branch] gh/tugsbayasgalan/33/orig -> origin/gh/tugsbayasgalan/33/orig 2025-10-10T00:37:13.2412659Z * [new branch] gh/tugsbayasgalan/34/base -> origin/gh/tugsbayasgalan/34/base 2025-10-10T00:37:13.2412829Z * [new branch] gh/tugsbayasgalan/34/head -> origin/gh/tugsbayasgalan/34/head 2025-10-10T00:37:13.2413055Z * [new branch] gh/tugsbayasgalan/34/orig -> origin/gh/tugsbayasgalan/34/orig 2025-10-10T00:37:13.2413266Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-10-10T00:37:13.2413442Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-10-10T00:37:13.2413611Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-10-10T00:37:13.2413787Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-10-10T00:37:13.2413964Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-10-10T00:37:13.2414132Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-10-10T00:37:13.2420111Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-10-10T00:37:13.2420337Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-10-10T00:37:13.2420667Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-10-10T00:37:13.2420833Z * [new branch] gh/tugsbayasgalan/38/base -> origin/gh/tugsbayasgalan/38/base 2025-10-10T00:37:13.2421152Z * [new branch] gh/tugsbayasgalan/38/head -> origin/gh/tugsbayasgalan/38/head 2025-10-10T00:37:13.2421316Z * [new branch] gh/tugsbayasgalan/38/orig -> origin/gh/tugsbayasgalan/38/orig 2025-10-10T00:37:13.2423519Z * [new branch] gh/tugsbayasgalan/39/base -> origin/gh/tugsbayasgalan/39/base 2025-10-10T00:37:13.2423743Z * [new branch] gh/tugsbayasgalan/39/head -> origin/gh/tugsbayasgalan/39/head 2025-10-10T00:37:13.2423916Z * [new branch] gh/tugsbayasgalan/39/orig -> origin/gh/tugsbayasgalan/39/orig 2025-10-10T00:37:13.2424095Z * [new branch] gh/tugsbayasgalan/40/base -> origin/gh/tugsbayasgalan/40/base 2025-10-10T00:37:13.2424288Z * [new branch] gh/tugsbayasgalan/40/head -> origin/gh/tugsbayasgalan/40/head 2025-10-10T00:37:13.2424494Z * [new branch] gh/tugsbayasgalan/40/orig -> origin/gh/tugsbayasgalan/40/orig 2025-10-10T00:37:13.2424671Z * [new branch] gh/tugsbayasgalan/41/base -> origin/gh/tugsbayasgalan/41/base 2025-10-10T00:37:13.2425053Z * [new branch] gh/tugsbayasgalan/41/head -> origin/gh/tugsbayasgalan/41/head 2025-10-10T00:37:13.2425596Z * [new branch] gh/tugsbayasgalan/41/orig -> origin/gh/tugsbayasgalan/41/orig 2025-10-10T00:37:13.2433745Z * [new branch] gh/tugsbayasgalan/42/base -> origin/gh/tugsbayasgalan/42/base 2025-10-10T00:37:13.2434108Z * [new branch] gh/tugsbayasgalan/42/head -> origin/gh/tugsbayasgalan/42/head 2025-10-10T00:37:13.2434354Z * [new branch] gh/tugsbayasgalan/42/orig -> origin/gh/tugsbayasgalan/42/orig 2025-10-10T00:37:13.2434531Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-10-10T00:37:13.2434807Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-10-10T00:37:13.2435513Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-10-10T00:37:13.2435707Z * [new branch] gh/tugsbayasgalan/44/base -> origin/gh/tugsbayasgalan/44/base 2025-10-10T00:37:13.2435881Z * [new branch] gh/tugsbayasgalan/44/head -> origin/gh/tugsbayasgalan/44/head 2025-10-10T00:37:13.2436040Z * [new branch] gh/tugsbayasgalan/44/orig -> origin/gh/tugsbayasgalan/44/orig 2025-10-10T00:37:13.2436221Z * [new branch] gh/tugsbayasgalan/45/base -> origin/gh/tugsbayasgalan/45/base 2025-10-10T00:37:13.2436374Z * [new branch] gh/tugsbayasgalan/45/head -> origin/gh/tugsbayasgalan/45/head 2025-10-10T00:37:13.2436534Z * [new branch] gh/tugsbayasgalan/45/orig -> origin/gh/tugsbayasgalan/45/orig 2025-10-10T00:37:13.2442406Z * [new branch] gh/tugsbayasgalan/46/base -> origin/gh/tugsbayasgalan/46/base 2025-10-10T00:37:13.2447286Z * [new branch] gh/tugsbayasgalan/46/head -> origin/gh/tugsbayasgalan/46/head 2025-10-10T00:37:13.2451722Z * [new branch] gh/tugsbayasgalan/46/orig -> origin/gh/tugsbayasgalan/46/orig 2025-10-10T00:37:13.2451935Z * [new branch] gh/tugsbayasgalan/47/base -> origin/gh/tugsbayasgalan/47/base 2025-10-10T00:37:13.2452121Z * [new branch] gh/tugsbayasgalan/47/head -> origin/gh/tugsbayasgalan/47/head 2025-10-10T00:37:13.2452314Z * [new branch] gh/tugsbayasgalan/47/orig -> origin/gh/tugsbayasgalan/47/orig 2025-10-10T00:37:13.2452493Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-10-10T00:37:13.2452674Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-10-10T00:37:13.2452849Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-10-10T00:37:13.2453231Z * [new branch] gh/tugsbayasgalan/49/base -> origin/gh/tugsbayasgalan/49/base 2025-10-10T00:37:13.2453415Z * [new branch] gh/tugsbayasgalan/49/head -> origin/gh/tugsbayasgalan/49/head 2025-10-10T00:37:13.2453592Z * [new branch] gh/tugsbayasgalan/49/orig -> origin/gh/tugsbayasgalan/49/orig 2025-10-10T00:37:13.2453769Z * [new branch] gh/tugsbayasgalan/50/base -> origin/gh/tugsbayasgalan/50/base 2025-10-10T00:37:13.2453987Z * [new branch] gh/tugsbayasgalan/50/head -> origin/gh/tugsbayasgalan/50/head 2025-10-10T00:37:13.2454172Z * [new branch] gh/tugsbayasgalan/50/orig -> origin/gh/tugsbayasgalan/50/orig 2025-10-10T00:37:13.2454344Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-10-10T00:37:13.2454516Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-10-10T00:37:13.2454676Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-10-10T00:37:13.2454849Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-10-10T00:37:13.2455026Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-10-10T00:37:13.2455205Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-10-10T00:37:13.2455377Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-10-10T00:37:13.2455602Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-10-10T00:37:13.2455778Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-10-10T00:37:13.2455946Z * [new branch] gh/tugsbayasgalan/54/base -> origin/gh/tugsbayasgalan/54/base 2025-10-10T00:37:13.2456118Z * [new branch] gh/tugsbayasgalan/54/head -> origin/gh/tugsbayasgalan/54/head 2025-10-10T00:37:13.2456278Z * [new branch] gh/tugsbayasgalan/54/orig -> origin/gh/tugsbayasgalan/54/orig 2025-10-10T00:37:13.2461997Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-10-10T00:37:13.2462346Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-10-10T00:37:13.2462589Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-10-10T00:37:13.2462800Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-10-10T00:37:13.2463040Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-10-10T00:37:13.2463210Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-10-10T00:37:13.2463368Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-10-10T00:37:13.2463531Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-10-10T00:37:13.2463698Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-10-10T00:37:13.2463863Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-10-10T00:37:13.2464035Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-10-10T00:37:13.2464202Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-10-10T00:37:13.2464356Z * [new branch] gh/v0i0/10/base -> origin/gh/v0i0/10/base 2025-10-10T00:37:13.2464501Z * [new branch] gh/v0i0/10/head -> origin/gh/v0i0/10/head 2025-10-10T00:37:13.2464638Z * [new branch] gh/v0i0/10/orig -> origin/gh/v0i0/10/orig 2025-10-10T00:37:13.2465685Z * [new branch] gh/v0i0/11/base -> origin/gh/v0i0/11/base 2025-10-10T00:37:13.2466094Z * [new branch] gh/v0i0/11/head -> origin/gh/v0i0/11/head 2025-10-10T00:37:13.2466758Z * [new branch] gh/v0i0/11/orig -> origin/gh/v0i0/11/orig 2025-10-10T00:37:13.2469019Z * [new branch] gh/v0i0/12/base -> origin/gh/v0i0/12/base 2025-10-10T00:37:13.2469332Z * [new branch] gh/v0i0/12/head -> origin/gh/v0i0/12/head 2025-10-10T00:37:13.2469475Z * [new branch] gh/v0i0/12/orig -> origin/gh/v0i0/12/orig 2025-10-10T00:37:13.2471222Z * [new branch] gh/v0i0/13/base -> origin/gh/v0i0/13/base 2025-10-10T00:37:13.2471560Z * [new branch] gh/v0i0/13/head -> origin/gh/v0i0/13/head 2025-10-10T00:37:13.2471705Z * [new branch] gh/v0i0/13/orig -> origin/gh/v0i0/13/orig 2025-10-10T00:37:13.2474437Z * [new branch] gh/v0i0/7/base -> origin/gh/v0i0/7/base 2025-10-10T00:37:13.2474765Z * [new branch] gh/v0i0/7/head -> origin/gh/v0i0/7/head 2025-10-10T00:37:13.2474930Z * [new branch] gh/v0i0/7/orig -> origin/gh/v0i0/7/orig 2025-10-10T00:37:13.2475226Z * [new branch] gh/v0i0/8/base -> origin/gh/v0i0/8/base 2025-10-10T00:37:13.2479276Z * [new branch] gh/v0i0/8/head -> origin/gh/v0i0/8/head 2025-10-10T00:37:13.2479591Z * [new branch] gh/v0i0/8/orig -> origin/gh/v0i0/8/orig 2025-10-10T00:37:13.2479906Z * [new branch] gh/v0i0/9/base -> origin/gh/v0i0/9/base 2025-10-10T00:37:13.2480159Z * [new branch] gh/v0i0/9/head -> origin/gh/v0i0/9/head 2025-10-10T00:37:13.2480295Z * [new branch] gh/v0i0/9/orig -> origin/gh/v0i0/9/orig 2025-10-10T00:37:13.2480695Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-10-10T00:37:13.2481631Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-10-10T00:37:13.2484732Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-10-10T00:37:13.2485077Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-10-10T00:37:13.2485341Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-10-10T00:37:13.2485613Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-10-10T00:37:13.2486047Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-10-10T00:37:13.2486577Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-10-10T00:37:13.2491053Z * [new branch] gh/vkuzo/7/base -> origin/gh/vkuzo/7/base 2025-10-10T00:37:13.2491373Z * [new branch] gh/vkuzo/7/head -> origin/gh/vkuzo/7/head 2025-10-10T00:37:13.2491605Z * [new branch] gh/vkuzo/7/orig -> origin/gh/vkuzo/7/orig 2025-10-10T00:37:13.2491777Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-10-10T00:37:13.2492032Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-10-10T00:37:13.2492180Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-10-10T00:37:13.2492756Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-10-10T00:37:13.2493081Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-10-10T00:37:13.2496496Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-10-10T00:37:13.2496828Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-10-10T00:37:13.2497140Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-10-10T00:37:13.2497442Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-10-10T00:37:13.2497601Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-10-10T00:37:13.2497745Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-10-10T00:37:13.2498250Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-10-10T00:37:13.2499982Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-10-10T00:37:13.2500499Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-10-10T00:37:13.2500671Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-10-10T00:37:13.2501782Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-10-10T00:37:13.2502161Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-10-10T00:37:13.2505463Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-10-10T00:37:13.2505714Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-10-10T00:37:13.2505875Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-10-10T00:37:13.2506045Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-10-10T00:37:13.2506403Z * [new branch] gh/weifengpy/31/base -> origin/gh/weifengpy/31/base 2025-10-10T00:37:13.2511217Z * [new branch] gh/weifengpy/31/head -> origin/gh/weifengpy/31/head 2025-10-10T00:37:13.2511398Z * [new branch] gh/weifengpy/31/orig -> origin/gh/weifengpy/31/orig 2025-10-10T00:37:13.2511548Z * [new branch] gh/weifengpy/32/base -> origin/gh/weifengpy/32/base 2025-10-10T00:37:13.2511769Z * [new branch] gh/weifengpy/32/head -> origin/gh/weifengpy/32/head 2025-10-10T00:37:13.2517801Z * [new branch] gh/weifengpy/32/orig -> origin/gh/weifengpy/32/orig 2025-10-10T00:37:13.2522767Z * [new branch] gh/weifengpy/33/base -> origin/gh/weifengpy/33/base 2025-10-10T00:37:13.2522960Z * [new branch] gh/weifengpy/33/head -> origin/gh/weifengpy/33/head 2025-10-10T00:37:13.2523120Z * [new branch] gh/weifengpy/33/orig -> origin/gh/weifengpy/33/orig 2025-10-10T00:37:13.2523288Z * [new branch] gh/weifengpy/34/base -> origin/gh/weifengpy/34/base 2025-10-10T00:37:13.2523443Z * [new branch] gh/weifengpy/34/head -> origin/gh/weifengpy/34/head 2025-10-10T00:37:13.2523590Z * [new branch] gh/weifengpy/34/orig -> origin/gh/weifengpy/34/orig 2025-10-10T00:37:13.2523740Z * [new branch] gh/weifengpy/35/base -> origin/gh/weifengpy/35/base 2025-10-10T00:37:13.2523911Z * [new branch] gh/weifengpy/35/head -> origin/gh/weifengpy/35/head 2025-10-10T00:37:13.2524060Z * [new branch] gh/weifengpy/35/orig -> origin/gh/weifengpy/35/orig 2025-10-10T00:37:13.2524209Z * [new branch] gh/weifengpy/36/base -> origin/gh/weifengpy/36/base 2025-10-10T00:37:13.2524349Z * [new branch] gh/weifengpy/36/head -> origin/gh/weifengpy/36/head 2025-10-10T00:37:13.2524491Z * [new branch] gh/weifengpy/36/orig -> origin/gh/weifengpy/36/orig 2025-10-10T00:37:13.2524650Z * [new branch] gh/weifengpy/37/base -> origin/gh/weifengpy/37/base 2025-10-10T00:37:13.2524792Z * [new branch] gh/weifengpy/37/head -> origin/gh/weifengpy/37/head 2025-10-10T00:37:13.2524939Z * [new branch] gh/weifengpy/37/orig -> origin/gh/weifengpy/37/orig 2025-10-10T00:37:13.2525106Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-10-10T00:37:13.2525437Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-10-10T00:37:13.2525597Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-10-10T00:37:13.2531638Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-10-10T00:37:13.2531874Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-10-10T00:37:13.2532053Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-10-10T00:37:13.2532221Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-10-10T00:37:13.2532378Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-10-10T00:37:13.2532542Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-10-10T00:37:13.2532702Z * [new branch] gh/williamwen42/281/base -> origin/gh/williamwen42/281/base 2025-10-10T00:37:13.2532863Z * [new branch] gh/williamwen42/281/head -> origin/gh/williamwen42/281/head 2025-10-10T00:37:13.2533019Z * [new branch] gh/williamwen42/281/orig -> origin/gh/williamwen42/281/orig 2025-10-10T00:37:13.2533175Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-10-10T00:37:13.2540549Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-10-10T00:37:13.2541151Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-10-10T00:37:13.2541464Z * [new branch] gh/williamwen42/285/base -> origin/gh/williamwen42/285/base 2025-10-10T00:37:13.2541715Z * [new branch] gh/williamwen42/285/head -> origin/gh/williamwen42/285/head 2025-10-10T00:37:13.2541931Z * [new branch] gh/williamwen42/285/orig -> origin/gh/williamwen42/285/orig 2025-10-10T00:37:13.2542685Z * [new branch] gh/williamwen42/286/base -> origin/gh/williamwen42/286/base 2025-10-10T00:37:13.2542884Z * [new branch] gh/williamwen42/286/head -> origin/gh/williamwen42/286/head 2025-10-10T00:37:13.2543058Z * [new branch] gh/williamwen42/286/orig -> origin/gh/williamwen42/286/orig 2025-10-10T00:37:13.2543641Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-10-10T00:37:13.2544372Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-10-10T00:37:13.2548887Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-10-10T00:37:13.2549113Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-10-10T00:37:13.2549657Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-10-10T00:37:13.2549919Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-10-10T00:37:13.2550100Z * [new branch] gh/williamwen42/289/base -> origin/gh/williamwen42/289/base 2025-10-10T00:37:13.2550263Z * [new branch] gh/williamwen42/289/head -> origin/gh/williamwen42/289/head 2025-10-10T00:37:13.2550570Z * [new branch] gh/williamwen42/289/orig -> origin/gh/williamwen42/289/orig 2025-10-10T00:37:13.2551466Z * [new branch] gh/williamwen42/290/base -> origin/gh/williamwen42/290/base 2025-10-10T00:37:13.2551872Z * [new branch] gh/williamwen42/290/head -> origin/gh/williamwen42/290/head 2025-10-10T00:37:13.2554493Z * [new branch] gh/williamwen42/290/orig -> origin/gh/williamwen42/290/orig 2025-10-10T00:37:13.2554865Z * [new branch] gh/williamwen42/291/base -> origin/gh/williamwen42/291/base 2025-10-10T00:37:13.2555123Z * [new branch] gh/williamwen42/291/head -> origin/gh/williamwen42/291/head 2025-10-10T00:37:13.2555638Z * [new branch] gh/williamwen42/291/orig -> origin/gh/williamwen42/291/orig 2025-10-10T00:37:13.2556156Z * [new branch] gh/williamwen42/292/base -> origin/gh/williamwen42/292/base 2025-10-10T00:37:13.2556982Z * [new branch] gh/williamwen42/292/head -> origin/gh/williamwen42/292/head 2025-10-10T00:37:13.2561268Z * [new branch] gh/williamwen42/292/orig -> origin/gh/williamwen42/292/orig 2025-10-10T00:37:13.2561619Z * [new branch] gh/williamwen42/293/base -> origin/gh/williamwen42/293/base 2025-10-10T00:37:13.2561858Z * [new branch] gh/williamwen42/293/head -> origin/gh/williamwen42/293/head 2025-10-10T00:37:13.2562067Z * [new branch] gh/williamwen42/293/orig -> origin/gh/williamwen42/293/orig 2025-10-10T00:37:13.2562286Z * [new branch] gh/williamwen42/294/base -> origin/gh/williamwen42/294/base 2025-10-10T00:37:13.2562976Z * [new branch] gh/williamwen42/294/head -> origin/gh/williamwen42/294/head 2025-10-10T00:37:13.2563291Z * [new branch] gh/williamwen42/294/orig -> origin/gh/williamwen42/294/orig 2025-10-10T00:37:13.2563574Z * [new branch] gh/williamwen42/295/base -> origin/gh/williamwen42/295/base 2025-10-10T00:37:13.2563728Z * [new branch] gh/williamwen42/295/head -> origin/gh/williamwen42/295/head 2025-10-10T00:37:13.2569702Z * [new branch] gh/williamwen42/295/orig -> origin/gh/williamwen42/295/orig 2025-10-10T00:37:13.2575295Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-10-10T00:37:13.2576958Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-10-10T00:37:13.2577311Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-10-10T00:37:13.2577505Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-10-10T00:37:13.2577751Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-10-10T00:37:13.2577916Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-10-10T00:37:13.2578145Z * [new branch] gh/williamwen42/298/base -> origin/gh/williamwen42/298/base 2025-10-10T00:37:13.2578370Z * [new branch] gh/williamwen42/298/head -> origin/gh/williamwen42/298/head 2025-10-10T00:37:13.2578548Z * [new branch] gh/williamwen42/298/orig -> origin/gh/williamwen42/298/orig 2025-10-10T00:37:13.2578733Z * [new branch] gh/williamwen42/299/base -> origin/gh/williamwen42/299/base 2025-10-10T00:37:13.2578910Z * [new branch] gh/williamwen42/299/head -> origin/gh/williamwen42/299/head 2025-10-10T00:37:13.2579104Z * [new branch] gh/williamwen42/299/orig -> origin/gh/williamwen42/299/orig 2025-10-10T00:37:13.2579288Z * [new branch] gh/williamwen42/300/base -> origin/gh/williamwen42/300/base 2025-10-10T00:37:13.2579500Z * [new branch] gh/williamwen42/300/head -> origin/gh/williamwen42/300/head 2025-10-10T00:37:13.2580120Z * [new branch] gh/williamwen42/300/orig -> origin/gh/williamwen42/300/orig 2025-10-10T00:37:13.2580320Z * [new branch] gh/williamwen42/301/base -> origin/gh/williamwen42/301/base 2025-10-10T00:37:13.2580474Z * [new branch] gh/williamwen42/301/head -> origin/gh/williamwen42/301/head 2025-10-10T00:37:13.2580661Z * [new branch] gh/williamwen42/301/orig -> origin/gh/williamwen42/301/orig 2025-10-10T00:37:13.2580809Z * [new branch] gh/williamwen42/302/base -> origin/gh/williamwen42/302/base 2025-10-10T00:37:13.2580958Z * [new branch] gh/williamwen42/302/head -> origin/gh/williamwen42/302/head 2025-10-10T00:37:13.2581441Z * [new branch] gh/williamwen42/302/orig -> origin/gh/williamwen42/302/orig 2025-10-10T00:37:13.2582089Z * [new branch] gh/williamwen42/303/base -> origin/gh/williamwen42/303/base 2025-10-10T00:37:13.2582282Z * [new branch] gh/williamwen42/303/head -> origin/gh/williamwen42/303/head 2025-10-10T00:37:13.2582890Z * [new branch] gh/williamwen42/303/orig -> origin/gh/williamwen42/303/orig 2025-10-10T00:37:13.2588553Z * [new branch] gh/williamwen42/304/base -> origin/gh/williamwen42/304/base 2025-10-10T00:37:13.2588901Z * [new branch] gh/williamwen42/304/head -> origin/gh/williamwen42/304/head 2025-10-10T00:37:13.2589135Z * [new branch] gh/williamwen42/304/orig -> origin/gh/williamwen42/304/orig 2025-10-10T00:37:13.2589307Z * [new branch] gh/williamwen42/305/base -> origin/gh/williamwen42/305/base 2025-10-10T00:37:13.2589456Z * [new branch] gh/williamwen42/305/head -> origin/gh/williamwen42/305/head 2025-10-10T00:37:13.2589606Z * [new branch] gh/williamwen42/305/orig -> origin/gh/williamwen42/305/orig 2025-10-10T00:37:13.2589747Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-10-10T00:37:13.2589894Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-10-10T00:37:13.2590173Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-10-10T00:37:13.2590462Z * [new branch] gh/williamwen42/307/base -> origin/gh/williamwen42/307/base 2025-10-10T00:37:13.2590899Z * [new branch] gh/williamwen42/307/head -> origin/gh/williamwen42/307/head 2025-10-10T00:37:13.2592944Z * [new branch] gh/williamwen42/307/orig -> origin/gh/williamwen42/307/orig 2025-10-10T00:37:13.2593285Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-10-10T00:37:13.2593526Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-10-10T00:37:13.2596625Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-10-10T00:37:13.2596938Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-10-10T00:37:13.2597672Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-10-10T00:37:13.2598198Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-10-10T00:37:13.2598380Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-10-10T00:37:13.2598524Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-10-10T00:37:13.2598703Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-10-10T00:37:13.2600395Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-10-10T00:37:13.2600616Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-10-10T00:37:13.2604560Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-10-10T00:37:13.2604893Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-10-10T00:37:13.2605121Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-10-10T00:37:13.2605282Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-10-10T00:37:13.2605559Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-10-10T00:37:13.2605754Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-10-10T00:37:13.2606366Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-10-10T00:37:13.2606542Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-10-10T00:37:13.2611136Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-10-10T00:37:13.2611465Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-10-10T00:37:13.2611691Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-10-10T00:37:13.2611909Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-10-10T00:37:13.2612066Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-10-10T00:37:13.2612243Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-10-10T00:37:13.2612444Z * [new branch] gh/xmfan/281/base -> origin/gh/xmfan/281/base 2025-10-10T00:37:13.2612984Z * [new branch] gh/xmfan/281/head -> origin/gh/xmfan/281/head 2025-10-10T00:37:13.2613653Z * [new branch] gh/xmfan/281/orig -> origin/gh/xmfan/281/orig 2025-10-10T00:37:13.2617065Z * [new branch] gh/xmfan/284/base -> origin/gh/xmfan/284/base 2025-10-10T00:37:13.2617387Z * [new branch] gh/xmfan/284/head -> origin/gh/xmfan/284/head 2025-10-10T00:37:13.2617604Z * [new branch] gh/xmfan/284/orig -> origin/gh/xmfan/284/orig 2025-10-10T00:37:13.2617815Z * [new branch] gh/xmfan/285/base -> origin/gh/xmfan/285/base 2025-10-10T00:37:13.2617966Z * [new branch] gh/xmfan/285/head -> origin/gh/xmfan/285/head 2025-10-10T00:37:13.2618792Z * [new branch] gh/xmfan/285/orig -> origin/gh/xmfan/285/orig 2025-10-10T00:37:13.2619357Z * [new branch] gh/xmfan/286/base -> origin/gh/xmfan/286/base 2025-10-10T00:37:13.2620148Z * [new branch] gh/xmfan/286/head -> origin/gh/xmfan/286/head 2025-10-10T00:37:13.2620835Z * [new branch] gh/xmfan/286/orig -> origin/gh/xmfan/286/orig 2025-10-10T00:37:13.2625000Z * [new branch] gh/xmfan/287/base -> origin/gh/xmfan/287/base 2025-10-10T00:37:13.2625164Z * [new branch] gh/xmfan/287/head -> origin/gh/xmfan/287/head 2025-10-10T00:37:13.2625305Z * [new branch] gh/xmfan/287/orig -> origin/gh/xmfan/287/orig 2025-10-10T00:37:13.2625433Z * [new branch] gh/xmfan/288/base -> origin/gh/xmfan/288/base 2025-10-10T00:37:13.2625567Z * [new branch] gh/xmfan/288/head -> origin/gh/xmfan/288/head 2025-10-10T00:37:13.2625728Z * [new branch] gh/xmfan/288/orig -> origin/gh/xmfan/288/orig 2025-10-10T00:37:13.2628083Z * [new branch] gh/xmfan/289/base -> origin/gh/xmfan/289/base 2025-10-10T00:37:13.2633313Z * [new branch] gh/xmfan/289/head -> origin/gh/xmfan/289/head 2025-10-10T00:37:13.2635496Z * [new branch] gh/xmfan/289/orig -> origin/gh/xmfan/289/orig 2025-10-10T00:37:13.2635699Z * [new branch] gh/xmfan/290/base -> origin/gh/xmfan/290/base 2025-10-10T00:37:13.2636273Z * [new branch] gh/xmfan/290/head -> origin/gh/xmfan/290/head 2025-10-10T00:37:13.2636435Z * [new branch] gh/xmfan/290/orig -> origin/gh/xmfan/290/orig 2025-10-10T00:37:13.2636748Z * [new branch] gh/xmfan/291/base -> origin/gh/xmfan/291/base 2025-10-10T00:37:13.2636900Z * [new branch] gh/xmfan/291/head -> origin/gh/xmfan/291/head 2025-10-10T00:37:13.2637040Z * [new branch] gh/xmfan/291/orig -> origin/gh/xmfan/291/orig 2025-10-10T00:37:13.2637172Z * [new branch] gh/xmfan/292/base -> origin/gh/xmfan/292/base 2025-10-10T00:37:13.2637300Z * [new branch] gh/xmfan/292/head -> origin/gh/xmfan/292/head 2025-10-10T00:37:13.2637436Z * [new branch] gh/xmfan/292/orig -> origin/gh/xmfan/292/orig 2025-10-10T00:37:13.2637740Z * [new branch] gh/xmfan/293/base -> origin/gh/xmfan/293/base 2025-10-10T00:37:13.2637868Z * [new branch] gh/xmfan/293/head -> origin/gh/xmfan/293/head 2025-10-10T00:37:13.2643037Z * [new branch] gh/xmfan/293/orig -> origin/gh/xmfan/293/orig 2025-10-10T00:37:13.2648667Z * [new branch] gh/xmfan/294/base -> origin/gh/xmfan/294/base 2025-10-10T00:37:13.2650725Z * [new branch] gh/xmfan/294/head -> origin/gh/xmfan/294/head 2025-10-10T00:37:13.2651001Z * [new branch] gh/xmfan/294/orig -> origin/gh/xmfan/294/orig 2025-10-10T00:37:13.2653805Z * [new branch] gh/xmfan/295/base -> origin/gh/xmfan/295/base 2025-10-10T00:37:13.2654089Z * [new branch] gh/xmfan/295/head -> origin/gh/xmfan/295/head 2025-10-10T00:37:13.2659365Z * [new branch] gh/xmfan/295/orig -> origin/gh/xmfan/295/orig 2025-10-10T00:37:13.2663810Z * [new branch] gh/xmfan/296/base -> origin/gh/xmfan/296/base 2025-10-10T00:37:13.2663996Z * [new branch] gh/xmfan/296/head -> origin/gh/xmfan/296/head 2025-10-10T00:37:13.2664160Z * [new branch] gh/xmfan/296/orig -> origin/gh/xmfan/296/orig 2025-10-10T00:37:13.2664306Z * [new branch] gh/xmfan/297/base -> origin/gh/xmfan/297/base 2025-10-10T00:37:13.2664461Z * [new branch] gh/xmfan/297/head -> origin/gh/xmfan/297/head 2025-10-10T00:37:13.2664808Z * [new branch] gh/xmfan/297/orig -> origin/gh/xmfan/297/orig 2025-10-10T00:37:13.2664965Z * [new branch] gh/xmfan/298/base -> origin/gh/xmfan/298/base 2025-10-10T00:37:13.2665122Z * [new branch] gh/xmfan/298/head -> origin/gh/xmfan/298/head 2025-10-10T00:37:13.2665271Z * [new branch] gh/xmfan/298/orig -> origin/gh/xmfan/298/orig 2025-10-10T00:37:13.2665413Z * [new branch] gh/xmfan/299/base -> origin/gh/xmfan/299/base 2025-10-10T00:37:13.2665561Z * [new branch] gh/xmfan/299/head -> origin/gh/xmfan/299/head 2025-10-10T00:37:13.2665718Z * [new branch] gh/xmfan/299/orig -> origin/gh/xmfan/299/orig 2025-10-10T00:37:13.2665865Z * [new branch] gh/xmfan/300/base -> origin/gh/xmfan/300/base 2025-10-10T00:37:13.2666014Z * [new branch] gh/xmfan/300/head -> origin/gh/xmfan/300/head 2025-10-10T00:37:13.2666165Z * [new branch] gh/xmfan/300/orig -> origin/gh/xmfan/300/orig 2025-10-10T00:37:13.2666554Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-10-10T00:37:13.2666718Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-10-10T00:37:13.2666853Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-10-10T00:37:13.2667001Z * [new branch] gh/xmfan/302/base -> origin/gh/xmfan/302/base 2025-10-10T00:37:13.2667151Z * [new branch] gh/xmfan/302/head -> origin/gh/xmfan/302/head 2025-10-10T00:37:13.2667302Z * [new branch] gh/xmfan/302/orig -> origin/gh/xmfan/302/orig 2025-10-10T00:37:13.2667450Z * [new branch] gh/xmfan/303/base -> origin/gh/xmfan/303/base 2025-10-10T00:37:13.2667594Z * [new branch] gh/xmfan/303/head -> origin/gh/xmfan/303/head 2025-10-10T00:37:13.2667748Z * [new branch] gh/xmfan/303/orig -> origin/gh/xmfan/303/orig 2025-10-10T00:37:13.2667879Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-10-10T00:37:13.2668026Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-10-10T00:37:13.2668173Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-10-10T00:37:13.2668411Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-10-10T00:37:13.2668569Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-10-10T00:37:13.2668718Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-10-10T00:37:13.2673399Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-10-10T00:37:13.2678984Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-10-10T00:37:13.2684599Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-10-10T00:37:13.2686749Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-10-10T00:37:13.2687089Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-10-10T00:37:13.2693273Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-10-10T00:37:13.2696474Z * [new branch] gh/xuanzhang816/25/base -> origin/gh/xuanzhang816/25/base 2025-10-10T00:37:13.2696690Z * [new branch] gh/xuanzhang816/25/head -> origin/gh/xuanzhang816/25/head 2025-10-10T00:37:13.2696861Z * [new branch] gh/xuanzhang816/25/orig -> origin/gh/xuanzhang816/25/orig 2025-10-10T00:37:13.2697024Z * [new branch] gh/xuanzhang816/26/base -> origin/gh/xuanzhang816/26/base 2025-10-10T00:37:13.2697355Z * [new branch] gh/xuanzhang816/26/head -> origin/gh/xuanzhang816/26/head 2025-10-10T00:37:13.2697529Z * [new branch] gh/xuanzhang816/26/orig -> origin/gh/xuanzhang816/26/orig 2025-10-10T00:37:13.2697700Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-10-10T00:37:13.2697867Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-10-10T00:37:13.2698044Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-10-10T00:37:13.2698196Z * [new branch] gh/xuanzhang816/28/base -> origin/gh/xuanzhang816/28/base 2025-10-10T00:37:13.2698354Z * [new branch] gh/xuanzhang816/28/head -> origin/gh/xuanzhang816/28/head 2025-10-10T00:37:13.2698513Z * [new branch] gh/xuanzhang816/28/orig -> origin/gh/xuanzhang816/28/orig 2025-10-10T00:37:13.2698678Z * [new branch] gh/xuanzhang816/29/base -> origin/gh/xuanzhang816/29/base 2025-10-10T00:37:13.2698861Z * [new branch] gh/xuanzhang816/29/head -> origin/gh/xuanzhang816/29/head 2025-10-10T00:37:13.2699008Z * [new branch] gh/xuanzhang816/29/orig -> origin/gh/xuanzhang816/29/orig 2025-10-10T00:37:13.2699179Z * [new branch] gh/xuanzhang816/30/base -> origin/gh/xuanzhang816/30/base 2025-10-10T00:37:13.2699342Z * [new branch] gh/xuanzhang816/30/head -> origin/gh/xuanzhang816/30/head 2025-10-10T00:37:13.2699562Z * [new branch] gh/xuanzhang816/30/orig -> origin/gh/xuanzhang816/30/orig 2025-10-10T00:37:13.2699729Z * [new branch] gh/xuanzhang816/31/base -> origin/gh/xuanzhang816/31/base 2025-10-10T00:37:13.2699894Z * [new branch] gh/xuanzhang816/31/head -> origin/gh/xuanzhang816/31/head 2025-10-10T00:37:13.2700052Z * [new branch] gh/xuanzhang816/31/orig -> origin/gh/xuanzhang816/31/orig 2025-10-10T00:37:13.2700215Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-10-10T00:37:13.2700367Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-10-10T00:37:13.2700529Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-10-10T00:37:13.2700694Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-10-10T00:37:13.2700922Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-10-10T00:37:13.2701085Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-10-10T00:37:13.2701249Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-10-10T00:37:13.2701409Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-10-10T00:37:13.2701589Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-10-10T00:37:13.2701794Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-10-10T00:37:13.2701939Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-10-10T00:37:13.2702088Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-10-10T00:37:13.2702249Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-10-10T00:37:13.2702393Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-10-10T00:37:13.2702546Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-10-10T00:37:13.2702718Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-10-10T00:37:13.2702870Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-10-10T00:37:13.2703009Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-10-10T00:37:13.2703201Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-10-10T00:37:13.2703342Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-10-10T00:37:13.2703488Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-10-10T00:37:13.2703901Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-10-10T00:37:13.2707805Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-10-10T00:37:13.2708011Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-10-10T00:37:13.2708175Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-10-10T00:37:13.2708333Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-10-10T00:37:13.2708508Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-10-10T00:37:13.2708749Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-10-10T00:37:13.2710872Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-10-10T00:37:13.2711147Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-10-10T00:37:13.2716654Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-10-10T00:37:13.2720230Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-10-10T00:37:13.2725707Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-10-10T00:37:13.2730008Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-10-10T00:37:13.2732006Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-10-10T00:37:13.2732186Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-10-10T00:37:13.2732333Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-10-10T00:37:13.2732468Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-10-10T00:37:13.2732610Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-10-10T00:37:13.2732939Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-10-10T00:37:13.2733074Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-10-10T00:37:13.2733217Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-10-10T00:37:13.2733350Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-10-10T00:37:13.2733490Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-10-10T00:37:13.2733626Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-10-10T00:37:13.2733765Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-10-10T00:37:13.2733897Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-10-10T00:37:13.2734041Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-10-10T00:37:13.2734182Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-10-10T00:37:13.2734314Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-10-10T00:37:13.2734456Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-10-10T00:37:13.2734589Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-10-10T00:37:13.2734736Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-10-10T00:37:13.2734917Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-10-10T00:37:13.2735046Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-10-10T00:37:13.2735180Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-10-10T00:37:13.2735305Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-10-10T00:37:13.2741587Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-10-10T00:37:13.2741898Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-10-10T00:37:13.2742056Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-10-10T00:37:13.2742271Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-10-10T00:37:13.2742556Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-10-10T00:37:13.2742789Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-10-10T00:37:13.2743394Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-10-10T00:37:13.2743707Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-10-10T00:37:13.2743903Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-10-10T00:37:13.2744199Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-10-10T00:37:13.2745313Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-10-10T00:37:13.2745777Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-10-10T00:37:13.2746827Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-10-10T00:37:13.2747254Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-10-10T00:37:13.2749138Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-10-10T00:37:13.2749511Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-10-10T00:37:13.2750549Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-10-10T00:37:13.2751071Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-10-10T00:37:13.2754283Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-10-10T00:37:13.2754459Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-10-10T00:37:13.2754590Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-10-10T00:37:13.2754723Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-10-10T00:37:13.2754876Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-10-10T00:37:13.2756729Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-10-10T00:37:13.2760573Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-10-10T00:37:13.2760740Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-10-10T00:37:13.2761189Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-10-10T00:37:13.2761338Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-10-10T00:37:13.2761483Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-10-10T00:37:13.2761616Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-10-10T00:37:13.2769695Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-10-10T00:37:13.2777343Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-10-10T00:37:13.2777561Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-10-10T00:37:13.2777698Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-10-10T00:37:13.2777832Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-10-10T00:37:13.2777982Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-10-10T00:37:13.2778108Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-10-10T00:37:13.2778244Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-10-10T00:37:13.2778390Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-10-10T00:37:13.2778525Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-10-10T00:37:13.2778653Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-10-10T00:37:13.2778781Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-10-10T00:37:13.2778912Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-10-10T00:37:13.2779037Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-10-10T00:37:13.2779170Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-10-10T00:37:13.2779297Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-10-10T00:37:13.2779426Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-10-10T00:37:13.2779549Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-10-10T00:37:13.2779674Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-10-10T00:37:13.2779809Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-10-10T00:37:13.2779934Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-10-10T00:37:13.2780065Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-10-10T00:37:13.2780192Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-10-10T00:37:13.2780366Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-10-10T00:37:13.2780499Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-10-10T00:37:13.2780625Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-10-10T00:37:13.2780762Z * [new branch] gh/ydwu4/318/base -> origin/gh/ydwu4/318/base 2025-10-10T00:37:13.2780887Z * [new branch] gh/ydwu4/318/head -> origin/gh/ydwu4/318/head 2025-10-10T00:37:13.2781018Z * [new branch] gh/ydwu4/318/orig -> origin/gh/ydwu4/318/orig 2025-10-10T00:37:13.2781375Z * [new branch] gh/ydwu4/319/base -> origin/gh/ydwu4/319/base 2025-10-10T00:37:13.2782403Z * [new branch] gh/ydwu4/319/head -> origin/gh/ydwu4/319/head 2025-10-10T00:37:13.2782887Z * [new branch] gh/ydwu4/319/orig -> origin/gh/ydwu4/319/orig 2025-10-10T00:37:13.2784495Z * [new branch] gh/ydwu4/320/base -> origin/gh/ydwu4/320/base 2025-10-10T00:37:13.2784764Z * [new branch] gh/ydwu4/320/head -> origin/gh/ydwu4/320/head 2025-10-10T00:37:13.2785753Z * [new branch] gh/ydwu4/320/orig -> origin/gh/ydwu4/320/orig 2025-10-10T00:37:13.2786835Z * [new branch] gh/ydwu4/321/base -> origin/gh/ydwu4/321/base 2025-10-10T00:37:13.2791030Z * [new branch] gh/ydwu4/321/head -> origin/gh/ydwu4/321/head 2025-10-10T00:37:13.2791349Z * [new branch] gh/ydwu4/321/orig -> origin/gh/ydwu4/321/orig 2025-10-10T00:37:13.2791491Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-10-10T00:37:13.2791640Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-10-10T00:37:13.2791764Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-10-10T00:37:13.2791904Z * [new branch] gh/ydwu4/324/base -> origin/gh/ydwu4/324/base 2025-10-10T00:37:13.2792220Z * [new branch] gh/ydwu4/324/head -> origin/gh/ydwu4/324/head 2025-10-10T00:37:13.2794082Z * [new branch] gh/ydwu4/324/orig -> origin/gh/ydwu4/324/orig 2025-10-10T00:37:13.2794350Z * [new branch] gh/ydwu4/325/base -> origin/gh/ydwu4/325/base 2025-10-10T00:37:13.2794507Z * [new branch] gh/ydwu4/325/head -> origin/gh/ydwu4/325/head 2025-10-10T00:37:13.2797765Z * [new branch] gh/ydwu4/325/orig -> origin/gh/ydwu4/325/orig 2025-10-10T00:37:13.2798096Z * [new branch] gh/ydwu4/326/base -> origin/gh/ydwu4/326/base 2025-10-10T00:37:13.2798265Z * [new branch] gh/ydwu4/326/head -> origin/gh/ydwu4/326/head 2025-10-10T00:37:13.2798427Z * [new branch] gh/ydwu4/326/orig -> origin/gh/ydwu4/326/orig 2025-10-10T00:37:13.2798604Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-10-10T00:37:13.2799316Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-10-10T00:37:13.2799670Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-10-10T00:37:13.2803707Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-10-10T00:37:13.2804039Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-10-10T00:37:13.2804480Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-10-10T00:37:13.2804669Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-10-10T00:37:13.2804813Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-10-10T00:37:13.2804950Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-10-10T00:37:13.2806829Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-10-10T00:37:13.2807146Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-10-10T00:37:13.2807313Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-10-10T00:37:13.2810465Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-10-10T00:37:13.2810791Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-10-10T00:37:13.2810954Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-10-10T00:37:13.2811091Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-10-10T00:37:13.2811298Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-10-10T00:37:13.2811618Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-10-10T00:37:13.2812115Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-10-10T00:37:13.2814876Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-10-10T00:37:13.2815052Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-10-10T00:37:13.2815185Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-10-10T00:37:13.2815327Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-10-10T00:37:13.2816512Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-10-10T00:37:13.2816843Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-10-10T00:37:13.2817497Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-10-10T00:37:13.2820556Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-10-10T00:37:13.2820773Z * [new branch] gh/ydwu4/336/base -> origin/gh/ydwu4/336/base 2025-10-10T00:37:13.2820926Z * [new branch] gh/ydwu4/336/head -> origin/gh/ydwu4/336/head 2025-10-10T00:37:13.2821077Z * [new branch] gh/ydwu4/336/orig -> origin/gh/ydwu4/336/orig 2025-10-10T00:37:13.2821268Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-10-10T00:37:13.2822313Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-10-10T00:37:13.2822591Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-10-10T00:37:13.2824200Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-10-10T00:37:13.2824489Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-10-10T00:37:13.2825788Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-10-10T00:37:13.2826078Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-10-10T00:37:13.2833427Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-10-10T00:37:13.2838704Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-10-10T00:37:13.2840548Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-10-10T00:37:13.2840691Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-10-10T00:37:13.2840833Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-10-10T00:37:13.2840962Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-10-10T00:37:13.2841109Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-10-10T00:37:13.2841237Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-10-10T00:37:13.2841539Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-10-10T00:37:13.2841673Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-10-10T00:37:13.2841879Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-10-10T00:37:13.2847311Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-10-10T00:37:13.2849413Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-10-10T00:37:13.2849680Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-10-10T00:37:13.2855114Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-10-10T00:37:13.2859302Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-10-10T00:37:13.2861687Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-10-10T00:37:13.2861886Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-10-10T00:37:13.2862187Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-10-10T00:37:13.2862362Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-10-10T00:37:13.2862498Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-10-10T00:37:13.2862638Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-10-10T00:37:13.2862970Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-10-10T00:37:13.2863121Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-10-10T00:37:13.2863256Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-10-10T00:37:13.2863388Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-10-10T00:37:13.2863535Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-10-10T00:37:13.2863667Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-10-10T00:37:13.2863816Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-10-10T00:37:13.2863963Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-10-10T00:37:13.2864115Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-10-10T00:37:13.2864247Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-10-10T00:37:13.2864391Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-10-10T00:37:13.2864536Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-10-10T00:37:13.2864681Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-10-10T00:37:13.2864826Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-10-10T00:37:13.2864976Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-10-10T00:37:13.2865130Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-10-10T00:37:13.2865293Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-10-10T00:37:13.2865448Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-10-10T00:37:13.2865599Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-10-10T00:37:13.2865746Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-10-10T00:37:13.2865892Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-10-10T00:37:13.2866024Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-10-10T00:37:13.2866235Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-10-10T00:37:13.2866542Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-10-10T00:37:13.2866681Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-10-10T00:37:13.2866838Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-10-10T00:37:13.2866988Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-10-10T00:37:13.2867140Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-10-10T00:37:13.2867290Z * [new branch] gh/zou3519/1195/base -> origin/gh/zou3519/1195/base 2025-10-10T00:37:13.2867441Z * [new branch] gh/zou3519/1195/head -> origin/gh/zou3519/1195/head 2025-10-10T00:37:13.2867575Z * [new branch] gh/zou3519/1195/orig -> origin/gh/zou3519/1195/orig 2025-10-10T00:37:13.2867696Z * [new branch] gh/zou3519/1196/base -> origin/gh/zou3519/1196/base 2025-10-10T00:37:13.2867977Z * [new branch] gh/zou3519/1196/head -> origin/gh/zou3519/1196/head 2025-10-10T00:37:13.2868385Z * [new branch] gh/zou3519/1196/orig -> origin/gh/zou3519/1196/orig 2025-10-10T00:37:13.2870834Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-10-10T00:37:13.2871218Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-10-10T00:37:13.2871379Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-10-10T00:37:13.2871582Z * [new branch] gh/zou3519/1198/base -> origin/gh/zou3519/1198/base 2025-10-10T00:37:13.2873042Z * [new branch] gh/zou3519/1198/head -> origin/gh/zou3519/1198/head 2025-10-10T00:37:13.2873411Z * [new branch] gh/zou3519/1198/orig -> origin/gh/zou3519/1198/orig 2025-10-10T00:37:13.2873807Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-10-10T00:37:13.2876239Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-10-10T00:37:13.2876412Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-10-10T00:37:13.2876548Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-10-10T00:37:13.2876999Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-10-10T00:37:13.2877585Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-10-10T00:37:13.2878744Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-10-10T00:37:13.2879106Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-10-10T00:37:13.2880043Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-10-10T00:37:13.2881141Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-10-10T00:37:13.2881419Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-10-10T00:37:13.2882732Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-10-10T00:37:13.2883052Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-10-10T00:37:13.2884038Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-10-10T00:37:13.2885747Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-10-10T00:37:13.2886032Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-10-10T00:37:13.2886468Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-10-10T00:37:13.2889557Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-10-10T00:37:13.2889895Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-10-10T00:37:13.2890052Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-10-10T00:37:13.2890458Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-10-10T00:37:13.2890610Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-10-10T00:37:13.2891553Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-10-10T00:37:13.2892572Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-10-10T00:37:13.2895310Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-10-10T00:37:13.2896038Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-10-10T00:37:13.2896417Z * [new branch] gh/zpcore/16/base -> origin/gh/zpcore/16/base 2025-10-10T00:37:13.2896547Z * [new branch] gh/zpcore/16/head -> origin/gh/zpcore/16/head 2025-10-10T00:37:13.2896678Z * [new branch] gh/zpcore/16/orig -> origin/gh/zpcore/16/orig 2025-10-10T00:37:13.2896798Z * [new branch] gh/zpcore/17/base -> origin/gh/zpcore/17/base 2025-10-10T00:37:13.2897521Z * [new branch] gh/zpcore/17/head -> origin/gh/zpcore/17/head 2025-10-10T00:37:13.2898053Z * [new branch] gh/zpcore/17/orig -> origin/gh/zpcore/17/orig 2025-10-10T00:37:13.2899207Z * [new branch] gh/zpcore/18/base -> origin/gh/zpcore/18/base 2025-10-10T00:37:13.2899440Z * [new branch] gh/zpcore/18/head -> origin/gh/zpcore/18/head 2025-10-10T00:37:13.2900434Z * [new branch] gh/zpcore/18/orig -> origin/gh/zpcore/18/orig 2025-10-10T00:37:13.2901369Z * [new branch] gh/zpcore/19/base -> origin/gh/zpcore/19/base 2025-10-10T00:37:13.2901905Z * [new branch] gh/zpcore/19/head -> origin/gh/zpcore/19/head 2025-10-10T00:37:13.2902794Z * [new branch] gh/zpcore/19/orig -> origin/gh/zpcore/19/orig 2025-10-10T00:37:13.2904045Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-10-10T00:37:13.2904377Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-10-10T00:37:13.2905870Z * [new branch] gh/zpcore/20/base -> origin/gh/zpcore/20/base 2025-10-10T00:37:13.2906233Z * [new branch] gh/zpcore/20/head -> origin/gh/zpcore/20/head 2025-10-10T00:37:13.2907282Z * [new branch] gh/zpcore/20/orig -> origin/gh/zpcore/20/orig 2025-10-10T00:37:13.2913412Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-10-10T00:37:13.2913576Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-10-10T00:37:13.2913725Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-10-10T00:37:13.2914267Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-10-10T00:37:13.2914443Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-10-10T00:37:13.2914583Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-10-10T00:37:13.2914734Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-10-10T00:37:13.2914861Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-10-10T00:37:13.2914997Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-10-10T00:37:13.2915125Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-10-10T00:37:13.2918799Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-10-10T00:37:13.2919033Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-10-10T00:37:13.2919664Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-10-10T00:37:13.2919838Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-10-10T00:37:13.2919977Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-10-10T00:37:13.2920133Z * [new branch] google-main -> origin/google-main 2025-10-10T00:37:13.2920277Z * [new branch] greencontext -> origin/greencontext 2025-10-10T00:37:13.2923884Z * [new branch] guangyey/config -> origin/guangyey/config 2025-10-10T00:37:13.2924184Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-10-10T00:37:13.2924349Z * [new branch] guangyey/reimport -> origin/guangyey/reimport 2025-10-10T00:37:13.2924499Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-10-10T00:37:13.2924746Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-10-10T00:37:13.2924939Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-10-10T00:37:13.2925195Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-10-10T00:37:13.2932380Z * [new branch] hc_baseline -> origin/hc_baseline 2025-10-10T00:37:13.2932550Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-10-10T00:37:13.2932688Z * [new branch] hhh_rand -> origin/hhh_rand 2025-10-10T00:37:13.2932850Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-10-10T00:37:13.2933030Z * [new branch] huba/debug_mode -> origin/huba/debug_mode 2025-10-10T00:37:13.2933169Z * [new branch] huba/dtensor_equal -> origin/huba/dtensor_equal 2025-10-10T00:37:13.2933287Z * [new branch] huba/f1 -> origin/huba/f1 2025-10-10T00:37:13.2933432Z * [new branch] huba/local_tensor -> origin/huba/local_tensor 2025-10-10T00:37:13.2934641Z * [new branch] ideep-update -> origin/ideep-update 2025-10-10T00:37:13.2934866Z * [new branch] increase-asan-build-memory -> origin/increase-asan-build-memory 2025-10-10T00:37:13.2935089Z * [new branch] inductor-perf-increase-timeout -> origin/inductor-perf-increase-timeout 2025-10-10T00:37:13.2935259Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-10-10T00:37:13.2935391Z * [new branch] inline -> origin/inline 2025-10-10T00:37:13.2935520Z * [new branch] inlining -> origin/inlining 2025-10-10T00:37:13.2935659Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-10-10T00:37:13.2940352Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-10-10T00:37:13.2940681Z * [new branch] install_free_tensors -> origin/install_free_tensors 2025-10-10T00:37:13.2940832Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-10-10T00:37:13.2941011Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-10-10T00:37:13.2941419Z * [new branch] issue#58739 -> origin/issue#58739 2025-10-10T00:37:13.2941757Z * [new branch] issue-161010-dynamo-stride-clone -> origin/issue-161010-dynamo-stride-clone 2025-10-10T00:37:13.2942361Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-10-10T00:37:13.2942924Z * [new branch] jathu/sve -> origin/jathu/sve 2025-10-10T00:37:13.2947181Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-10-10T00:37:13.2949719Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-10-10T00:37:13.2950031Z * [new branch] jeanschmidt-patch-1 -> origin/jeanschmidt-patch-1 2025-10-10T00:37:13.2956389Z * [new branch] jerryzh168-patch-1 -> origin/jerryzh168-patch-1 2025-10-10T00:37:13.2958562Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-10-10T00:37:13.2963368Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-10-10T00:37:13.2966016Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-10-10T00:37:13.2966215Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-10-10T00:37:13.2966408Z * [new branch] justinchu/allowlist-api-onnx -> origin/justinchu/allowlist-api-onnx 2025-10-10T00:37:13.2966690Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-10-10T00:37:13.2970811Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-10-10T00:37:13.2975629Z * [new branch] justinchuby/typo-error -> origin/justinchuby/typo-error 2025-10-10T00:37:13.2978804Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-10-10T00:37:13.2982073Z * [new branch] kainan_test -> origin/kainan_test 2025-10-10T00:37:13.2982448Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-10-10T00:37:13.2982730Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-10-10T00:37:13.2982999Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-10-10T00:37:13.2983268Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-10-10T00:37:13.2984096Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-10-10T00:37:13.2984280Z * [new branch] libtorch_free_so -> origin/libtorch_free_so 2025-10-10T00:37:13.2984595Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-10-10T00:37:13.2984794Z * [new branch] llama4-stable -> origin/llama4-stable 2025-10-10T00:37:13.2984920Z * [new branch] logdetfix -> origin/logdetfix 2025-10-10T00:37:13.2985047Z * [new branch] logsumexp -> origin/logsumexp 2025-10-10T00:37:13.2985186Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-10-10T00:37:13.2985335Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-10-10T00:37:13.2985519Z * [new branch] lucaskabela/cherrypick_163769 -> origin/lucaskabela/cherrypick_163769 2025-10-10T00:37:13.2985677Z * [new branch] lucaskabela/fix_164814 -> origin/lucaskabela/fix_164814 2025-10-10T00:37:13.2985827Z * [new branch] lucaskabela/fix_164823 -> origin/lucaskabela/fix_164823 2025-10-10T00:37:13.2985980Z * [new branch] lucaskabela/fix_164875 -> origin/lucaskabela/fix_164875 2025-10-10T00:37:13.2986157Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-10-10T00:37:13.2986397Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-10-10T00:37:13.2986594Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-10-10T00:37:13.2986980Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-10-10T00:37:13.2987203Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-10-10T00:37:13.2987443Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-10-10T00:37:13.2987601Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-10-10T00:37:13.2987786Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-10-10T00:37:13.2987908Z * [new branch] main -> origin/main 2025-10-10T00:37:13.2988144Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-10-10T00:37:13.2988340Z * [new branch] main-enable-b200-symm-mem-test -> origin/main-enable-b200-symm-mem-test 2025-10-10T00:37:13.2988496Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-10-10T00:37:13.2988639Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-10-10T00:37:13.2988778Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-10-10T00:37:13.2988917Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-10-10T00:37:13.2989050Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-10-10T00:37:13.2989230Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-10-10T00:37:13.2989365Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-10-10T00:37:13.2989496Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-10-10T00:37:13.2989634Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-10-10T00:37:13.2989769Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-10-10T00:37:13.2990094Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-10-10T00:37:13.2990277Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-10-10T00:37:13.2990495Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-10-10T00:37:13.2990700Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-10-10T00:37:13.2990854Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-10-10T00:37:13.2991046Z * [new branch] masnesral/pt2_internal_logging -> origin/masnesral/pt2_internal_logging 2025-10-10T00:37:13.2991193Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-10-10T00:37:13.2991352Z * [new branch] mingw_constant_buffer -> origin/mingw_constant_buffer 2025-10-10T00:37:13.2991512Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-10-10T00:37:13.2991646Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-10-10T00:37:13.2991774Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-10-10T00:37:13.2991922Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-10-10T00:37:13.2992104Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-10-10T00:37:13.2992262Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-10-10T00:37:13.2992407Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-10-10T00:37:13.2992595Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-10-10T00:37:13.2992787Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-10-10T00:37:13.2993068Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-10-10T00:37:13.2993250Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-10-10T00:37:13.2993394Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-10-10T00:37:13.2993870Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-10-10T00:37:13.2995687Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-10-10T00:37:13.2995878Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-10-10T00:37:13.2996156Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-10-10T00:37:13.3001565Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-10-10T00:37:13.3003634Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-10-10T00:37:13.3003834Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-10-10T00:37:13.3004002Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-10-10T00:37:13.3004199Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-10-10T00:37:13.3004517Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-10-10T00:37:13.3004656Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-10-10T00:37:13.3004782Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-10-10T00:37:13.3004914Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-10-10T00:37:13.3005057Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-10-10T00:37:13.3005188Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-10-10T00:37:13.3011844Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-10-10T00:37:13.3017315Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-10-10T00:37:13.3022413Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-10-10T00:37:13.3022612Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-10-10T00:37:13.3022776Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-10-10T00:37:13.3022949Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-10-10T00:37:13.3023097Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-10-10T00:37:13.3023261Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-10-10T00:37:13.3023401Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-10-10T00:37:13.3023546Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-10-10T00:37:13.3023684Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-10-10T00:37:13.3023822Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-10-10T00:37:13.3023968Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-10-10T00:37:13.3024102Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-10-10T00:37:13.3024244Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-10-10T00:37:13.3024383Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-10-10T00:37:13.3024765Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-10-10T00:37:13.3024897Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-10-10T00:37:13.3025024Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-10-10T00:37:13.3025158Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-10-10T00:37:13.3025281Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-10-10T00:37:13.3025418Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-10-10T00:37:13.3025548Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-10-10T00:37:13.3025676Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-10-10T00:37:13.3025801Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-10-10T00:37:13.3025921Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-10-10T00:37:13.3026053Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-10-10T00:37:13.3026172Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-10-10T00:37:13.3026498Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-10-10T00:37:13.3026637Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-10-10T00:37:13.3026762Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-10-10T00:37:13.3026995Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-10-10T00:37:13.3027175Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-10-10T00:37:13.3027349Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-10-10T00:37:13.3027479Z * [new branch] mlazos/main -> origin/mlazos/main 2025-10-10T00:37:13.3027679Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-10-10T00:37:13.3031300Z * [new branch] mlazos/mark-static-update -> origin/mlazos/mark-static-update 2025-10-10T00:37:13.3031779Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-10-10T00:37:13.3037877Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-10-10T00:37:13.3042919Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-10-10T00:37:13.3048243Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-10-10T00:37:13.3050497Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-10-10T00:37:13.3055596Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-10-10T00:37:13.3060919Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-10-10T00:37:13.3062736Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-10-10T00:37:13.3062909Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-10-10T00:37:13.3063082Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-10-10T00:37:13.3063238Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-10-10T00:37:13.3063396Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-10-10T00:37:13.3063551Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-10-10T00:37:13.3063714Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-10-10T00:37:13.3063849Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-10-10T00:37:13.3063983Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-10-10T00:37:13.3064325Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-10-10T00:37:13.3064487Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-10-10T00:37:13.3064679Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-10-10T00:37:13.3064816Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-10-10T00:37:13.3064974Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-10-10T00:37:13.3065171Z * [new branch] mlazos/test -> origin/mlazos/test 2025-10-10T00:37:13.3065314Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-10-10T00:37:13.3065482Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-10-10T00:37:13.3065650Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-10-10T00:37:13.3065812Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-10-10T00:37:13.3065978Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-10-10T00:37:13.3066131Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-10-10T00:37:13.3066452Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-10-10T00:37:13.3066627Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-10-10T00:37:13.3066865Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-10-10T00:37:13.3067049Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-10-10T00:37:13.3067208Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-10-10T00:37:13.3067386Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-10-10T00:37:13.3067534Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-10-10T00:37:13.3067693Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-10-10T00:37:13.3067842Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-10-10T00:37:13.3067999Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-10-10T00:37:13.3068134Z * [new branch] module-shim -> origin/module-shim 2025-10-10T00:37:13.3068269Z * [new branch] module-stack -> origin/module-stack 2025-10-10T00:37:13.3068392Z * [new branch] more_ck_Fixes -> origin/more_ck_Fixes 2025-10-10T00:37:13.3068572Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-10-10T00:37:13.3068767Z * [new branch] move_aws_steps_inside_setup_rocm -> origin/move_aws_steps_inside_setup_rocm 2025-10-10T00:37:13.3068925Z * [new branch] msaroufim-patch-1 -> origin/msaroufim-patch-1 2025-10-10T00:37:13.3069056Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-10-10T00:37:13.3069206Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-10-10T00:37:13.3069339Z * [new branch] msaroufim/cub -> origin/msaroufim/cub 2025-10-10T00:37:13.3069548Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-10-10T00:37:13.3069683Z * [new branch] msaroufim/patchx -> origin/msaroufim/patchx 2025-10-10T00:37:13.3069833Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-10-10T00:37:13.3069970Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-10-10T00:37:13.3070204Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-10-10T00:37:13.3070338Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-10-10T00:37:13.3070583Z * [new branch] nWEIdia/skip-tests-for-pr-159494 -> origin/nWEIdia/skip-tests-for-pr-159494 2025-10-10T00:37:13.3070748Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-10-10T00:37:13.3070876Z * [new branch] new-codegen -> origin/new-codegen 2025-10-10T00:37:13.3071012Z * [new branch] newtest-base -> origin/newtest-base 2025-10-10T00:37:13.3071166Z * [new branch] ngimel/allgather_format -> origin/ngimel/allgather_format 2025-10-10T00:37:13.3071303Z * [new branch] ngimel/cat_perf2 -> origin/ngimel/cat_perf2 2025-10-10T00:37:13.3077397Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-10-10T00:37:13.3081002Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-10-10T00:37:13.3083259Z * [new branch] ngimel/scatter_add_multid -> origin/ngimel/scatter_add_multid 2025-10-10T00:37:13.3083516Z * [new branch] nightly -> origin/nightly 2025-10-10T00:37:13.3086956Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-10-10T00:37:13.3087336Z * [new branch] nikitaved/addmm_epilogue_fusions -> origin/nikitaved/addmm_epilogue_fusions 2025-10-10T00:37:13.3087825Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-10-10T00:37:13.3088171Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-10-10T00:37:13.3088850Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-10-10T00:37:13.3089099Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-10-10T00:37:13.3089290Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-10-10T00:37:13.3089467Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-10-10T00:37:13.3089616Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-10-10T00:37:13.3089782Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-10-10T00:37:13.3089919Z * [new branch] nofun-hack -> origin/nofun-hack 2025-10-10T00:37:13.3090073Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-10-10T00:37:13.3090229Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-10-10T00:37:13.3090380Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-10-10T00:37:13.3090528Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-10-10T00:37:13.3090675Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-10-10T00:37:13.3090822Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-10-10T00:37:13.3091311Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-10-10T00:37:13.3091577Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-10-10T00:37:13.3091736Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-10-10T00:37:13.3091964Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-10-10T00:37:13.3092116Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-10-10T00:37:13.3092337Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-10-10T00:37:13.3097845Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-10-10T00:37:13.3098054Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-10-10T00:37:13.3098211Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-10-10T00:37:13.3098458Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-10-10T00:37:13.3101827Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-10-10T00:37:13.3102505Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-10-10T00:37:13.3102694Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-10-10T00:37:13.3102840Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-10-10T00:37:13.3103023Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-10-10T00:37:13.3103199Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-10-10T00:37:13.3103365Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-10-10T00:37:13.3103509Z * [new branch] padded-tensor -> origin/padded-tensor 2025-10-10T00:37:13.3103641Z * [new branch] pca2 -> origin/pca2 2025-10-10T00:37:13.3103772Z * [new branch] perf_ops -> origin/perf_ops 2025-10-10T00:37:13.3104330Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-10-10T00:37:13.3104998Z * [new branch] perserve_node_meta_decomp -> origin/perserve_node_meta_decomp 2025-10-10T00:37:13.3106679Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-10-10T00:37:13.3111050Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-10-10T00:37:13.3111338Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-10-10T00:37:13.3111570Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-10-10T00:37:13.3111760Z * [new branch] pianpwk/base_view_shape_key -> origin/pianpwk/base_view_shape_key 2025-10-10T00:37:13.3111926Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-10-10T00:37:13.3112102Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-10-10T00:37:13.3112431Z * [new branch] pianpwk/debug_mode_inductor -> origin/pianpwk/debug_mode_inductor 2025-10-10T00:37:13.3113091Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-10-10T00:37:13.3113348Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-10-10T00:37:13.3114282Z * [new branch] pianpwk/debugmode_show_ids -> origin/pianpwk/debugmode_show_ids 2025-10-10T00:37:13.3114925Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-10-10T00:37:13.3115594Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-10-10T00:37:13.3116388Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-10-10T00:37:13.3117015Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-10-10T00:37:13.3117868Z * [new branch] pianpwk/dtensor_shape_metadata_guard -> origin/pianpwk/dtensor_shape_metadata_guard 2025-10-10T00:37:13.3118796Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-10-10T00:37:13.3119366Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-10-10T00:37:13.3120824Z * [new branch] pianpwk/multi_kernel_l1 -> origin/pianpwk/multi_kernel_l1 2025-10-10T00:37:13.3121030Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-10-10T00:37:13.3121987Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-10-10T00:37:13.3122714Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-10-10T00:37:13.3123009Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-10-10T00:37:13.3127907Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-10-10T00:37:13.3128133Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-10-10T00:37:13.3128305Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-10-10T00:37:13.3128553Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-10-10T00:37:13.3128746Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-10-10T00:37:13.3128931Z * [new branch] pianpwk/triton_benchmark_hints -> origin/pianpwk/triton_benchmark_hints 2025-10-10T00:37:13.3129107Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-10-10T00:37:13.3129406Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-10-10T00:37:13.3129646Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-10-10T00:37:13.3133511Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-10-10T00:37:13.3133757Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-10-10T00:37:13.3134046Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-10-10T00:37:13.3140696Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-10-10T00:37:13.3142774Z * [new branch] piz/add_wait -> origin/piz/add_wait 2025-10-10T00:37:13.3142967Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-10-10T00:37:13.3143139Z * [new branch] pool-separate -> origin/pool-separate 2025-10-10T00:37:13.3143266Z * [new branch] pr-156087 -> origin/pr-156087 2025-10-10T00:37:13.3143400Z * [new branch] pr/131860 -> origin/pr/131860 2025-10-10T00:37:13.3143544Z * [new branch] pre_compile_checks -> origin/pre_compile_checks 2025-10-10T00:37:13.3143689Z * [new branch] predispatch_to -> origin/predispatch_to 2025-10-10T00:37:13.3143914Z * [new branch] prepare-perf-baseline-number-2.8 -> origin/prepare-perf-baseline-number-2.8 2025-10-10T00:37:13.3144081Z * [new branch] prepare-perf-number-2.9 -> origin/prepare-perf-number-2.9 2025-10-10T00:37:13.3144234Z * [new branch] profiler-enabled -> origin/profiler-enabled 2025-10-10T00:37:13.3144369Z * [new branch] provenance_doc_2 -> origin/provenance_doc_2 2025-10-10T00:37:13.3148313Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-10-10T00:37:13.3153421Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-10-10T00:37:13.3155865Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-10-10T00:37:13.3162635Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-10-10T00:37:13.3164710Z * [new branch] quantile-docs -> origin/quantile-docs 2025-10-10T00:37:13.3164984Z * [new branch] quint-bits -> origin/quint-bits 2025-10-10T00:37:13.3171364Z * [new branch] reland-fx-annotate -> origin/reland-fx-annotate 2025-10-10T00:37:13.3176248Z * [new branch] reland_req_nvsh -> origin/reland_req_nvsh 2025-10-10T00:37:13.3178384Z * [new branch] release/1.10 -> origin/release/1.10 2025-10-10T00:37:13.3178829Z * [new branch] release/1.11 -> origin/release/1.11 2025-10-10T00:37:13.3179023Z * [new branch] release/1.12 -> origin/release/1.12 2025-10-10T00:37:13.3179143Z * [new branch] release/1.13 -> origin/release/1.13 2025-10-10T00:37:13.3179272Z * [new branch] release/1.4 -> origin/release/1.4 2025-10-10T00:37:13.3179396Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-10-10T00:37:13.3179526Z * [new branch] release/1.5 -> origin/release/1.5 2025-10-10T00:37:13.3179640Z * [new branch] release/1.6 -> origin/release/1.6 2025-10-10T00:37:13.3179753Z * [new branch] release/1.7 -> origin/release/1.7 2025-10-10T00:37:13.3179874Z * [new branch] release/1.8 -> origin/release/1.8 2025-10-10T00:37:13.3179988Z * [new branch] release/1.9 -> origin/release/1.9 2025-10-10T00:37:13.3180278Z * [new branch] release/2.0 -> origin/release/2.0 2025-10-10T00:37:13.3180398Z * [new branch] release/2.1 -> origin/release/2.1 2025-10-10T00:37:13.3180525Z * [new branch] release/2.2 -> origin/release/2.2 2025-10-10T00:37:13.3180644Z * [new branch] release/2.3 -> origin/release/2.3 2025-10-10T00:37:13.3180773Z * [new branch] release/2.4 -> origin/release/2.4 2025-10-10T00:37:13.3180893Z * [new branch] release/2.5 -> origin/release/2.5 2025-10-10T00:37:13.3181005Z * [new branch] release/2.6 -> origin/release/2.6 2025-10-10T00:37:13.3181126Z * [new branch] release/2.7 -> origin/release/2.7 2025-10-10T00:37:13.3181239Z * [new branch] release/2.8 -> origin/release/2.8 2025-10-10T00:37:13.3181351Z * [new branch] release/2.9 -> origin/release/2.9 2025-10-10T00:37:13.3181485Z * [new branch] release_notes -> origin/release_notes 2025-10-10T00:37:13.3181627Z * [new branch] remove_header_code -> origin/remove_header_code 2025-10-10T00:37:13.3181792Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-10-10T00:37:13.3181967Z * [new branch] repackage-vllm-nightlies -> origin/repackage-vllm-nightlies 2025-10-10T00:37:13.3182206Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-10-10T00:37:13.3182418Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-10-10T00:37:13.3182621Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-10-10T00:37:13.3182831Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-10-10T00:37:13.3183058Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-10-10T00:37:13.3183258Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-10-10T00:37:13.3183441Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-10-10T00:37:13.3183775Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-10-10T00:37:13.3184154Z * [new branch] revert-163802-camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 -> origin/revert-163802-camyll/cherrypick_3016616ccbba3dc9bb6a80eb4a81a846ddf49cc9 2025-10-10T00:37:13.3184329Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-10-10T00:37:13.3184494Z * [new branch] rocm-test-yml-update -> origin/rocm-test-yml-update 2025-10-10T00:37:13.3184616Z * [new branch] rocm_op_bench -> origin/rocm_op_bench 2025-10-10T00:37:13.3184763Z * [new branch] ruisi/aot_eager_pass -> origin/ruisi/aot_eager_pass 2025-10-10T00:37:13.3184907Z * [new branch] ruisi/placement_trace -> origin/ruisi/placement_trace 2025-10-10T00:37:13.3185184Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-10-10T00:37:13.3185443Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-10-10T00:37:13.3185602Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-10-10T00:37:13.3185728Z * [new branch] rzou/njt -> origin/rzou/njt 2025-10-10T00:37:13.3185851Z * [new branch] rzou/pca -> origin/rzou/pca 2025-10-10T00:37:13.3186667Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-10-10T00:37:13.3194632Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-10-10T00:37:13.3199763Z * [new branch] samplevllm -> origin/samplevllm 2025-10-10T00:37:13.3205436Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-10-10T00:37:13.3209635Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-10-10T00:37:13.3213874Z * [new branch] save -> origin/save 2025-10-10T00:37:13.3218409Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-10-10T00:37:13.3222948Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-10-10T00:37:13.3223202Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-10-10T00:37:13.3223372Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-10-10T00:37:13.3223532Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-10-10T00:37:13.3223694Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-10-10T00:37:13.3223921Z * [new branch] soulitzer/reland-codev-grad-dtype -> origin/soulitzer/reland-codev-grad-dtype 2025-10-10T00:37:13.3224091Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-10-10T00:37:13.3224246Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-10-10T00:37:13.3224388Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-10-10T00:37:13.3224571Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-10-10T00:37:13.3224721Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-10-10T00:37:13.3224846Z * [new branch] suo -> origin/suo 2025-10-10T00:37:13.3225015Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-10-10T00:37:13.3225147Z * [new branch] sve-poc -> origin/sve-poc 2025-10-10T00:37:13.3225500Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-10-10T00:37:13.3225642Z * [new branch] svekars-patch-2 -> origin/svekars-patch-2 2025-10-10T00:37:13.3225788Z * [new branch] svekars-patch-3 -> origin/svekars-patch-3 2025-10-10T00:37:13.3225924Z * [new branch] svekars-patch-4 -> origin/svekars-patch-4 2025-10-10T00:37:13.3226066Z * [new branch] svekars-patch-5 -> origin/svekars-patch-5 2025-10-10T00:37:13.3226203Z * [new branch] switch-bn -> origin/switch-bn 2025-10-10T00:37:13.3226580Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-10-10T00:37:13.3226760Z * [new branch] tenpercent/ck_rocm_ci_v3 -> origin/tenpercent/ck_rocm_ci_v3 2025-10-10T00:37:13.3226917Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-10-10T00:37:13.3227089Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-10-10T00:37:13.3227281Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-10-10T00:37:13.3227416Z * [new branch] test-old -> origin/test-old 2025-10-10T00:37:13.3227618Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-10-10T00:37:13.3227768Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-10-10T00:37:13.3227941Z * [new branch] test/inductor -> origin/test/inductor 2025-10-10T00:37:13.3228076Z * [new branch] test_quantization -> origin/test_quantization 2025-10-10T00:37:13.3228240Z * [new branch] tianren/customOp_autotune -> origin/tianren/customOp_autotune 2025-10-10T00:37:13.3228408Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-10-10T00:37:13.3228569Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-10-10T00:37:13.3228748Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-10-10T00:37:13.3228893Z * [new branch] tianren/remove_repeate -> origin/tianren/remove_repeate 2025-10-10T00:37:13.3229023Z * [new branch] tianren/test -> origin/tianren/test 2025-10-10T00:37:13.3229168Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-10-10T00:37:13.3229299Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-10-10T00:37:13.3229446Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-10-10T00:37:13.3229599Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-10-10T00:37:13.3229744Z * [new branch] transpose_pack_fusion -> origin/transpose_pack_fusion 2025-10-10T00:37:13.3229875Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-10-10T00:37:13.3230002Z * [new branch] triton_kernel -> origin/triton_kernel 2025-10-10T00:37:13.3230171Z * [new branch] trunk-tagging-multi-commits -> origin/trunk-tagging-multi-commits 2025-10-10T00:37:13.3230297Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-10-10T00:37:13.3230417Z * [new branch] type_dec -> origin/type_dec 2025-10-10T00:37:13.3230578Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-10-10T00:37:13.3230700Z * [new branch] unlift -> origin/unlift 2025-10-10T00:37:13.3230940Z * [new branch] update-audio-commit-hash/17567864209-1799-1 -> origin/update-audio-commit-hash/17567864209-1799-1 2025-10-10T00:37:13.3231206Z * [new branch] update-audio-commit-hash/17599208654-1801-1 -> origin/update-audio-commit-hash/17599208654-1801-1 2025-10-10T00:37:13.3231422Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-10-10T00:37:13.3231841Z * [new branch] update-audio-commit-hash/17657093113-1804-1 -> origin/update-audio-commit-hash/17657093113-1804-1 2025-10-10T00:37:13.3232072Z * [new branch] update-audio-commit-hash/17688961747-1806-1 -> origin/update-audio-commit-hash/17688961747-1806-1 2025-10-10T00:37:13.3234093Z * [new branch] update-audio-commit-hash/17703952853-1807-1 -> origin/update-audio-commit-hash/17703952853-1807-1 2025-10-10T00:37:13.3234745Z * [new branch] update-audio-commit-hash/18392707270-1874-1 -> origin/update-audio-commit-hash/18392707270-1874-1 2025-10-10T00:37:13.3234964Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-10-10T00:37:13.3235281Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-10-10T00:37:13.3235548Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-10-10T00:37:13.3241626Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-10-10T00:37:13.3247649Z * [new branch] update-vision-commit-hash/18361653903-1869-1 -> origin/update-vision-commit-hash/18361653903-1869-1 2025-10-10T00:37:13.3252664Z * [new branch] update-vllm-commit-hash/17536029887-1798-1 -> origin/update-vllm-commit-hash/17536029887-1798-1 2025-10-10T00:37:13.3256725Z * [new branch] update-vllm-commit-hash/17599208654-1801-1 -> origin/update-vllm-commit-hash/17599208654-1801-1 2025-10-10T00:37:13.3261108Z * [new branch] update-vllm-commit-hash/17657093113-1804-1 -> origin/update-vllm-commit-hash/17657093113-1804-1 2025-10-10T00:37:13.3261379Z * [new branch] update-vllm-commit-hash/17703952853-1807-1 -> origin/update-vllm-commit-hash/17703952853-1807-1 2025-10-10T00:37:13.3261610Z * [new branch] update-vllm-commit-hash/17718740812-1808-1 -> origin/update-vllm-commit-hash/17718740812-1808-1 2025-10-10T00:37:13.3261827Z * [new branch] update-vllm-commit-hash/17782703922-1813-1 -> origin/update-vllm-commit-hash/17782703922-1813-1 2025-10-10T00:37:13.3262066Z * [new branch] update-vllm-commit-hash/17814169036-1822-1 -> origin/update-vllm-commit-hash/17814169036-1822-1 2025-10-10T00:37:13.3262278Z * [new branch] update-vllm-commit-hash/17844794719-1823-1 -> origin/update-vllm-commit-hash/17844794719-1823-1 2025-10-10T00:37:13.3262495Z * [new branch] update-vllm-commit-hash/17872674059-1830-1 -> origin/update-vllm-commit-hash/17872674059-1830-1 2025-10-10T00:37:13.3262711Z * [new branch] update-vllm-commit-hash/17901034819-1833-1 -> origin/update-vllm-commit-hash/17901034819-1833-1 2025-10-10T00:37:13.3262922Z * [new branch] update-vllm-commit-hash/17932176396-1836-1 -> origin/update-vllm-commit-hash/17932176396-1836-1 2025-10-10T00:37:13.3263141Z * [new branch] update-vllm-commit-hash/17962545886-1842-1 -> origin/update-vllm-commit-hash/17962545886-1842-1 2025-10-10T00:37:13.3263357Z * [new branch] update-vllm-commit-hash/17993166855-1844-1 -> origin/update-vllm-commit-hash/17993166855-1844-1 2025-10-10T00:37:13.3263583Z * [new branch] update-vllm-commit-hash/18052321282-1848-1 -> origin/update-vllm-commit-hash/18052321282-1848-1 2025-10-10T00:37:13.3263845Z * [new branch] update-vllm-commit-hash/18066820738-1849-1 -> origin/update-vllm-commit-hash/18066820738-1849-1 2025-10-10T00:37:13.3264081Z * [new branch] update-vllm-commit-hash/18081987460-1850-1 -> origin/update-vllm-commit-hash/18081987460-1850-1 2025-10-10T00:37:13.3264533Z * [new branch] update-vllm-commit-hash/18114584510-1852-1 -> origin/update-vllm-commit-hash/18114584510-1852-1 2025-10-10T00:37:13.3264781Z * [new branch] update-vllm-commit-hash/18147226974-1853-1 -> origin/update-vllm-commit-hash/18147226974-1853-1 2025-10-10T00:37:13.3265004Z * [new branch] update-vllm-commit-hash/18236802781-1857-1 -> origin/update-vllm-commit-hash/18236802781-1857-1 2025-10-10T00:37:13.3265235Z * [new branch] update-xla-commit-hash/17725712604-203-1 -> origin/update-xla-commit-hash/17725712604-203-1 2025-10-10T00:37:13.3265479Z * [new branch] update-xla-commit-hash/17908176340-204-1 -> origin/update-xla-commit-hash/17908176340-204-1 2025-10-10T00:37:13.3265697Z * [new branch] update-xla-commit-hash/18273597034-206-1 -> origin/update-xla-commit-hash/18273597034-206-1 2025-10-10T00:37:13.3265934Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-10-10T00:37:13.3266095Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-10-10T00:37:13.3266556Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-10-10T00:37:13.3266733Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-10-10T00:37:13.3266888Z * [new branch] update_slow_tests_1757922057 -> origin/update_slow_tests_1757922057 2025-10-10T00:37:13.3267117Z * [new branch] update_slow_tests_1758526845 -> origin/update_slow_tests_1758526845 2025-10-10T00:37:13.3267275Z * [new branch] update_slow_tests_1759736444 -> origin/update_slow_tests_1759736444 2025-10-10T00:37:13.3267482Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-10-10T00:37:13.3267644Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-10-10T00:37:13.3267807Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-10-10T00:37:13.3267922Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-10-10T00:37:13.3268036Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-10-10T00:37:13.3268146Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-10-10T00:37:13.3268249Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-10-10T00:37:13.3268361Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-10-10T00:37:13.3268463Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-10-10T00:37:13.3268565Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-10-10T00:37:13.3268692Z * [new branch] validate_fn -> origin/validate_fn 2025-10-10T00:37:13.3268825Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-10-10T00:37:13.3268956Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-10-10T00:37:13.3269076Z * [new branch] varlen-api -> origin/varlen-api 2025-10-10T00:37:13.3269195Z * [new branch] varlen_api -> origin/varlen_api 2025-10-10T00:37:13.3269325Z * [new branch] viable/strict -> origin/viable/strict 2025-10-10T00:37:13.3269525Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-10-10T00:37:13.3269651Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-10-10T00:37:13.3269763Z * [new branch] vllmpin -> origin/vllmpin 2025-10-10T00:37:13.3269889Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-10-10T00:37:13.3270046Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-10-10T00:37:13.3270160Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-10-10T00:37:13.3270428Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-10-10T00:37:13.3275023Z * [new branch] whc/uneven -> origin/whc/uneven 2025-10-10T00:37:13.3279317Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-10-10T00:37:13.3284487Z * [new branch] williamwen42-patch-1 -> origin/williamwen42-patch-1 2025-10-10T00:37:13.3284809Z * [new branch] win_warnings -> origin/win_warnings 2025-10-10T00:37:13.3285020Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-10-10T00:37:13.3285220Z * [new branch] windows_mmap -> origin/windows_mmap 2025-10-10T00:37:13.3285393Z * [new branch] xmfan-war -> origin/xmfan-war 2025-10-10T00:37:13.3285556Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-10-10T00:37:13.3285698Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-10-10T00:37:13.3285959Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-10-10T00:37:13.3286104Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-10-10T00:37:13.3286368Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-10-10T00:37:13.3286497Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-10-10T00:37:13.3286630Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-10-10T00:37:13.3286752Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-10-10T00:37:13.3286915Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-10-10T00:37:13.3287062Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-10-10T00:37:13.3287205Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-10-10T00:37:13.3287352Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-10-10T00:37:13.3287504Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-10-10T00:37:13.3287645Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-10-10T00:37:13.3287774Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-10-10T00:37:13.3287919Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-10-10T00:37:13.3288050Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-10-10T00:37:13.3288177Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-10-10T00:37:13.3288321Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-10-10T00:37:13.3288447Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-10-10T00:37:13.3288719Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-10-10T00:37:13.3288905Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-10-10T00:37:13.3289197Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-10-10T00:37:13.3289422Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-10-10T00:37:13.3290755Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-10-10T00:37:13.3291055Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-10-10T00:37:13.3291573Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-10-10T00:37:13.3293606Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-10-10T00:37:13.3293933Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-10-10T00:37:13.3294175Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-10-10T00:37:13.3294366Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-10-10T00:37:13.3294864Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-10-10T00:37:13.3300134Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-10-10T00:37:13.3300594Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-10-10T00:37:13.3301031Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-10-10T00:37:13.3301324Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-10-10T00:37:13.3301507Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-10-10T00:37:13.3301640Z * [new branch] xmfan/test -> origin/xmfan/test 2025-10-10T00:37:13.3301840Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-10-10T00:37:13.3302162Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-10-10T00:37:13.3302365Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-10-10T00:37:13.3303392Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-10-10T00:37:13.3303575Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-10-10T00:37:13.3306778Z * [new branch] yiming/improve_sharding_error_msg -> origin/yiming/improve_sharding_error_msg 2025-10-10T00:37:13.3307170Z * [new branch] yiming/precompile_benchmark -> origin/yiming/precompile_benchmark 2025-10-10T00:37:13.3307416Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-10-10T00:37:13.3307686Z * [new branch] ysiraichi/install-fmtlib-headers-v12 -> origin/ysiraichi/install-fmtlib-headers-v12 2025-10-10T00:37:13.3308304Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-10-10T00:37:13.3308557Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-10-10T00:37:13.3309497Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-10-10T00:37:13.3309921Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-10-10T00:37:13.3310567Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-10-10T00:37:13.3311626Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-10-10T00:37:13.3312051Z * [new branch] zb2p -> origin/zb2p 2025-10-10T00:37:13.3315314Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-10-10T00:37:13.3315607Z * [new branch] zhxchen17/aot_compile_fix_load_guard_manager -> origin/zhxchen17/aot_compile_fix_load_guard_manager 2025-10-10T00:37:13.3315840Z * [new branch] zhxchen17/precompile/source_info -> origin/zhxchen17/precompile/source_info 2025-10-10T00:37:13.3316430Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-10-10T00:37:13.3320794Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-10-10T00:37:13.3321156Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-10-10T00:37:13.3321307Z * [new branch] zxiiro/c7i-docs -> origin/zxiiro/c7i-docs 2025-10-10T00:37:13.3321479Z * [new branch] zxiiro/c7i-linux-4xlarge -> origin/zxiiro/c7i-linux-4xlarge 2025-10-10T00:37:13.3321669Z * [new branch] zxiiro/c7i-linux-build-yaml -> origin/zxiiro/c7i-linux-build-yaml 2025-10-10T00:37:13.3321808Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-10-10T00:37:13.3322036Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-10-10T00:37:13.3323103Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-10-10T00:37:13.3323579Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-10-10T00:37:13.3324971Z * [new tag] ciflow/b200-symm-mem/163767 -> ciflow/b200-symm-mem/163767 2025-10-10T00:37:13.3325123Z * [new tag] ciflow/b200/163955 -> ciflow/b200/163955 2025-10-10T00:37:13.3325481Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-10-10T00:37:13.3325976Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-10-10T00:37:13.3329387Z * [new tag] ciflow/binaries/164769 -> ciflow/binaries/164769 2025-10-10T00:37:13.3329564Z * [new tag] ciflow/binaries/164894 -> ciflow/binaries/164894 2025-10-10T00:37:13.3329898Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-10-10T00:37:13.3330094Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-10-10T00:37:13.3330255Z * [new tag] ciflow/binaries_wheel/159104 -> ciflow/binaries_wheel/159104 2025-10-10T00:37:13.3330544Z * [new tag] ciflow/binaries_wheel/164935 -> ciflow/binaries_wheel/164935 2025-10-10T00:37:13.3330778Z * [new tag] ciflow/h100-cutlass-backend/163767 -> ciflow/h100-cutlass-backend/163767 2025-10-10T00:37:13.3330976Z * [new tag] ciflow/h100-cutlass-backend/164747 -> ciflow/h100-cutlass-backend/164747 2025-10-10T00:37:13.3331172Z * [new tag] ciflow/h100-distributed/163767 -> ciflow/h100-distributed/163767 2025-10-10T00:37:13.3331611Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-10-10T00:37:13.3332201Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-10-10T00:37:13.3332377Z * [new tag] ciflow/h100-symm-mem/163767 -> ciflow/h100-symm-mem/163767 2025-10-10T00:37:13.3333806Z * [new tag] ciflow/h100-symm-mem/164747 -> ciflow/h100-symm-mem/164747 2025-10-10T00:37:13.3334158Z * [new tag] ciflow/h100-symm-mem/164965 -> ciflow/h100-symm-mem/164965 2025-10-10T00:37:13.3334402Z * [new tag] ciflow/h100-symm-mem/165101 -> ciflow/h100-symm-mem/165101 2025-10-10T00:37:13.3334558Z * [new tag] ciflow/h100/163955 -> ciflow/h100/163955 2025-10-10T00:37:13.3334766Z * [new tag] ciflow/h100/164474 -> ciflow/h100/164474 2025-10-10T00:37:13.3334908Z * [new tag] ciflow/h100/164705 -> ciflow/h100/164705 2025-10-10T00:37:13.3335220Z * [new tag] ciflow/h100/164790 -> ciflow/h100/164790 2025-10-10T00:37:13.3336309Z * [new tag] ciflow/h100/164930 -> ciflow/h100/164930 2025-10-10T00:37:13.3336479Z * [new tag] ciflow/h100/165055 -> ciflow/h100/165055 2025-10-10T00:37:13.3337011Z * [new tag] ciflow/inductor-micro-benchmark/164747 -> ciflow/inductor-micro-benchmark/164747 2025-10-10T00:37:13.3337561Z * [new tag] ciflow/inductor-perf-compare/163767 -> ciflow/inductor-perf-compare/163767 2025-10-10T00:37:13.3338014Z * [new tag] ciflow/inductor-perf-compare/164747 -> ciflow/inductor-perf-compare/164747 2025-10-10T00:37:13.3339685Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/151845 -> ciflow/inductor-perf-test-nightly-rocm/151845 2025-10-10T00:37:13.3339970Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/164747 -> ciflow/inductor-perf-test-nightly-rocm/164747 2025-10-10T00:37:13.3340256Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/161512 -> ciflow/inductor-perf-test-nightly-x86-zen/161512 2025-10-10T00:37:13.3340681Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/162954 -> ciflow/inductor-perf-test-nightly-x86-zen/162954 2025-10-10T00:37:13.3341046Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/163767 -> ciflow/inductor-perf-test-nightly-x86-zen/163767 2025-10-10T00:37:13.3341406Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/164126 -> ciflow/inductor-perf-test-nightly-x86-zen/164126 2025-10-10T00:37:13.3342182Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/164747 -> ciflow/inductor-perf-test-nightly-x86-zen/164747 2025-10-10T00:37:13.3342554Z * [new tag] ciflow/inductor-periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> ciflow/inductor-periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T00:37:13.3342857Z * [new tag] ciflow/inductor-periodic/156592 -> ciflow/inductor-periodic/156592 2025-10-10T00:37:13.3343586Z * [new tag] ciflow/inductor-periodic/164492 -> ciflow/inductor-periodic/164492 2025-10-10T00:37:13.3344002Z * [new tag] ciflow/inductor-periodic/73adac05d13babb75410c3e033fdce57aa16881a -> ciflow/inductor-periodic/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T00:37:13.3344333Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-10-10T00:37:13.3347352Z * [new tag] ciflow/inductor-rocm/161280 -> ciflow/inductor-rocm/161280 2025-10-10T00:37:13.3347707Z * [new tag] ciflow/inductor-rocm/162478 -> ciflow/inductor-rocm/162478 2025-10-10T00:37:13.3347932Z * [new tag] ciflow/inductor-rocm/163767 -> ciflow/inductor-rocm/163767 2025-10-10T00:37:13.3348106Z * [new tag] ciflow/inductor-rocm/164618 -> ciflow/inductor-rocm/164618 2025-10-10T00:37:13.3348276Z * [new tag] ciflow/inductor-rocm/164747 -> ciflow/inductor-rocm/164747 2025-10-10T00:37:13.3348427Z * [new tag] ciflow/inductor-rocm/164769 -> ciflow/inductor-rocm/164769 2025-10-10T00:37:13.3348661Z * [new tag] ciflow/inductor-rocm/165080 -> ciflow/inductor-rocm/165080 2025-10-10T00:37:13.3349321Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-10-10T00:37:13.3349473Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-10-10T00:37:13.3349630Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-10-10T00:37:13.3350430Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-10-10T00:37:13.3350582Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-10-10T00:37:13.3353626Z * [new tag] ciflow/inductor/149003 -> ciflow/inductor/149003 2025-10-10T00:37:13.3353940Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-10-10T00:37:13.3354107Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-10-10T00:37:13.3354237Z * [new tag] ciflow/inductor/156592 -> ciflow/inductor/156592 2025-10-10T00:37:13.3354356Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-10-10T00:37:13.3354474Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-10-10T00:37:13.3354883Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-10-10T00:37:13.3355567Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-10-10T00:37:13.3355732Z * [new tag] ciflow/inductor/158872 -> ciflow/inductor/158872 2025-10-10T00:37:13.3355867Z * [new tag] ciflow/inductor/159523 -> ciflow/inductor/159523 2025-10-10T00:37:13.3355999Z * [new tag] ciflow/inductor/160266 -> ciflow/inductor/160266 2025-10-10T00:37:13.3357026Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-10-10T00:37:13.3357184Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-10-10T00:37:13.3360262Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-10-10T00:37:13.3360572Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-10-10T00:37:13.3360741Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-10-10T00:37:13.3360917Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-10-10T00:37:13.3361055Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-10-10T00:37:13.3361183Z * [new tag] ciflow/inductor/160611 -> ciflow/inductor/160611 2025-10-10T00:37:13.3361392Z * [new tag] ciflow/inductor/160843 -> ciflow/inductor/160843 2025-10-10T00:37:13.3362048Z * [new tag] ciflow/inductor/160903 -> ciflow/inductor/160903 2025-10-10T00:37:13.3362485Z * [new tag] ciflow/inductor/161118 -> ciflow/inductor/161118 2025-10-10T00:37:13.3362854Z * [new tag] ciflow/inductor/161158 -> ciflow/inductor/161158 2025-10-10T00:37:13.3363229Z * [new tag] ciflow/inductor/161280 -> ciflow/inductor/161280 2025-10-10T00:37:13.3366158Z * [new tag] ciflow/inductor/161320 -> ciflow/inductor/161320 2025-10-10T00:37:13.3366473Z * [new tag] ciflow/inductor/161485 -> ciflow/inductor/161485 2025-10-10T00:37:13.3366642Z * [new tag] ciflow/inductor/161495 -> ciflow/inductor/161495 2025-10-10T00:37:13.3366760Z * [new tag] ciflow/inductor/161512 -> ciflow/inductor/161512 2025-10-10T00:37:13.3366893Z * [new tag] ciflow/inductor/162031 -> ciflow/inductor/162031 2025-10-10T00:37:13.3367143Z * [new tag] ciflow/inductor/162066 -> ciflow/inductor/162066 2025-10-10T00:37:13.3367307Z * [new tag] ciflow/inductor/162294 -> ciflow/inductor/162294 2025-10-10T00:37:13.3367443Z * [new tag] ciflow/inductor/162340 -> ciflow/inductor/162340 2025-10-10T00:37:13.3367926Z * [new tag] ciflow/inductor/162470 -> ciflow/inductor/162470 2025-10-10T00:37:13.3371316Z * [new tag] ciflow/inductor/162523 -> ciflow/inductor/162523 2025-10-10T00:37:13.3371643Z * [new tag] ciflow/inductor/162542 -> ciflow/inductor/162542 2025-10-10T00:37:13.3371807Z * [new tag] ciflow/inductor/162768 -> ciflow/inductor/162768 2025-10-10T00:37:13.3371987Z * [new tag] ciflow/inductor/162899 -> ciflow/inductor/162899 2025-10-10T00:37:13.3372144Z * [new tag] ciflow/inductor/162900 -> ciflow/inductor/162900 2025-10-10T00:37:13.3372285Z * [new tag] ciflow/inductor/162901 -> ciflow/inductor/162901 2025-10-10T00:37:13.3372491Z * [new tag] ciflow/inductor/162903 -> ciflow/inductor/162903 2025-10-10T00:37:13.3373188Z * [new tag] ciflow/inductor/162905 -> ciflow/inductor/162905 2025-10-10T00:37:13.3373341Z * [new tag] ciflow/inductor/162954 -> ciflow/inductor/162954 2025-10-10T00:37:13.3373619Z * [new tag] ciflow/inductor/162990 -> ciflow/inductor/162990 2025-10-10T00:37:13.3373890Z * [new tag] ciflow/inductor/163027 -> ciflow/inductor/163027 2025-10-10T00:37:13.3374023Z * [new tag] ciflow/inductor/163028 -> ciflow/inductor/163028 2025-10-10T00:37:13.3374292Z * [new tag] ciflow/inductor/163053 -> ciflow/inductor/163053 2025-10-10T00:37:13.3374733Z * [new tag] ciflow/inductor/163185 -> ciflow/inductor/163185 2025-10-10T00:37:13.3375194Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-10-10T00:37:13.3375639Z * [new tag] ciflow/inductor/163490 -> ciflow/inductor/163490 2025-10-10T00:37:13.3378321Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-10-10T00:37:13.3378811Z * [new tag] ciflow/inductor/163517 -> ciflow/inductor/163517 2025-10-10T00:37:13.3378968Z * [new tag] ciflow/inductor/163527 -> ciflow/inductor/163527 2025-10-10T00:37:13.3379127Z * [new tag] ciflow/inductor/163533 -> ciflow/inductor/163533 2025-10-10T00:37:13.3389398Z * [new tag] ciflow/inductor/163602 -> ciflow/inductor/163602 2025-10-10T00:37:13.3389735Z * [new tag] ciflow/inductor/163617 -> ciflow/inductor/163617 2025-10-10T00:37:13.3389907Z * [new tag] ciflow/inductor/163667 -> ciflow/inductor/163667 2025-10-10T00:37:13.3390091Z * [new tag] ciflow/inductor/163671 -> ciflow/inductor/163671 2025-10-10T00:37:13.3390556Z * [new tag] ciflow/inductor/163767 -> ciflow/inductor/163767 2025-10-10T00:37:13.3390807Z * [new tag] ciflow/inductor/163772 -> ciflow/inductor/163772 2025-10-10T00:37:13.3391463Z * [new tag] ciflow/inductor/163806 -> ciflow/inductor/163806 2025-10-10T00:37:13.3391643Z * [new tag] ciflow/inductor/163936 -> ciflow/inductor/163936 2025-10-10T00:37:13.3391802Z * [new tag] ciflow/inductor/163976 -> ciflow/inductor/163976 2025-10-10T00:37:13.3391930Z * [new tag] ciflow/inductor/164039 -> ciflow/inductor/164039 2025-10-10T00:37:13.3392055Z * [new tag] ciflow/inductor/164040 -> ciflow/inductor/164040 2025-10-10T00:37:13.3392192Z * [new tag] ciflow/inductor/164130 -> ciflow/inductor/164130 2025-10-10T00:37:13.3392321Z * [new tag] ciflow/inductor/164144 -> ciflow/inductor/164144 2025-10-10T00:37:13.3392456Z * [new tag] ciflow/inductor/164202 -> ciflow/inductor/164202 2025-10-10T00:37:13.3392581Z * [new tag] ciflow/inductor/164212 -> ciflow/inductor/164212 2025-10-10T00:37:13.3392714Z * [new tag] ciflow/inductor/164273 -> ciflow/inductor/164273 2025-10-10T00:37:13.3392839Z * [new tag] ciflow/inductor/164277 -> ciflow/inductor/164277 2025-10-10T00:37:13.3392972Z * [new tag] ciflow/inductor/164291 -> ciflow/inductor/164291 2025-10-10T00:37:13.3393104Z * [new tag] ciflow/inductor/164296 -> ciflow/inductor/164296 2025-10-10T00:37:13.3393229Z * [new tag] ciflow/inductor/164304 -> ciflow/inductor/164304 2025-10-10T00:37:13.3393362Z * [new tag] ciflow/inductor/164318 -> ciflow/inductor/164318 2025-10-10T00:37:13.3393486Z * [new tag] ciflow/inductor/164321 -> ciflow/inductor/164321 2025-10-10T00:37:13.3393611Z * [new tag] ciflow/inductor/164324 -> ciflow/inductor/164324 2025-10-10T00:37:13.3393745Z * [new tag] ciflow/inductor/164341 -> ciflow/inductor/164341 2025-10-10T00:37:13.3393872Z * [new tag] ciflow/inductor/164343 -> ciflow/inductor/164343 2025-10-10T00:37:13.3394000Z * [new tag] ciflow/inductor/164344 -> ciflow/inductor/164344 2025-10-10T00:37:13.3394123Z * [new tag] ciflow/inductor/164359 -> ciflow/inductor/164359 2025-10-10T00:37:13.3394408Z * [new tag] ciflow/inductor/164373 -> ciflow/inductor/164373 2025-10-10T00:37:13.3394532Z * [new tag] ciflow/inductor/164379 -> ciflow/inductor/164379 2025-10-10T00:37:13.3394655Z * [new tag] ciflow/inductor/164384 -> ciflow/inductor/164384 2025-10-10T00:37:13.3394789Z * [new tag] ciflow/inductor/164404 -> ciflow/inductor/164404 2025-10-10T00:37:13.3394919Z * [new tag] ciflow/inductor/164405 -> ciflow/inductor/164405 2025-10-10T00:37:13.3395049Z * [new tag] ciflow/inductor/164414 -> ciflow/inductor/164414 2025-10-10T00:37:13.3395164Z * [new tag] ciflow/inductor/164422 -> ciflow/inductor/164422 2025-10-10T00:37:13.3400044Z * [new tag] ciflow/inductor/164433 -> ciflow/inductor/164433 2025-10-10T00:37:13.3405486Z * [new tag] ciflow/inductor/164474 -> ciflow/inductor/164474 2025-10-10T00:37:13.3409997Z * [new tag] ciflow/inductor/164488 -> ciflow/inductor/164488 2025-10-10T00:37:13.3415001Z * [new tag] ciflow/inductor/164492 -> ciflow/inductor/164492 2025-10-10T00:37:13.3419404Z * [new tag] ciflow/inductor/164497 -> ciflow/inductor/164497 2025-10-10T00:37:13.3423688Z * [new tag] ciflow/inductor/164498 -> ciflow/inductor/164498 2025-10-10T00:37:13.3423865Z * [new tag] ciflow/inductor/164500 -> ciflow/inductor/164500 2025-10-10T00:37:13.3424203Z * [new tag] ciflow/inductor/164507 -> ciflow/inductor/164507 2025-10-10T00:37:13.3424342Z * [new tag] ciflow/inductor/164519 -> ciflow/inductor/164519 2025-10-10T00:37:13.3424535Z * [new tag] ciflow/inductor/164521 -> ciflow/inductor/164521 2025-10-10T00:37:13.3424668Z * [new tag] ciflow/inductor/164522 -> ciflow/inductor/164522 2025-10-10T00:37:13.3424809Z * [new tag] ciflow/inductor/164523 -> ciflow/inductor/164523 2025-10-10T00:37:13.3424934Z * [new tag] ciflow/inductor/164524 -> ciflow/inductor/164524 2025-10-10T00:37:13.3425068Z * [new tag] ciflow/inductor/164525 -> ciflow/inductor/164525 2025-10-10T00:37:13.3425193Z * [new tag] ciflow/inductor/164526 -> ciflow/inductor/164526 2025-10-10T00:37:13.3425327Z * [new tag] ciflow/inductor/164527 -> ciflow/inductor/164527 2025-10-10T00:37:13.3425456Z * [new tag] ciflow/inductor/164533 -> ciflow/inductor/164533 2025-10-10T00:37:13.3425588Z * [new tag] ciflow/inductor/164537 -> ciflow/inductor/164537 2025-10-10T00:37:13.3425715Z * [new tag] ciflow/inductor/164548 -> ciflow/inductor/164548 2025-10-10T00:37:13.3425839Z * [new tag] ciflow/inductor/164557 -> ciflow/inductor/164557 2025-10-10T00:37:13.3425981Z * [new tag] ciflow/inductor/164558 -> ciflow/inductor/164558 2025-10-10T00:37:13.3426107Z * [new tag] ciflow/inductor/164560 -> ciflow/inductor/164560 2025-10-10T00:37:13.3426240Z * [new tag] ciflow/inductor/164565 -> ciflow/inductor/164565 2025-10-10T00:37:13.3426551Z * [new tag] ciflow/inductor/164577 -> ciflow/inductor/164577 2025-10-10T00:37:13.3426680Z * [new tag] ciflow/inductor/164609 -> ciflow/inductor/164609 2025-10-10T00:37:13.3426817Z * [new tag] ciflow/inductor/164610 -> ciflow/inductor/164610 2025-10-10T00:37:13.3426942Z * [new tag] ciflow/inductor/164611 -> ciflow/inductor/164611 2025-10-10T00:37:13.3427074Z * [new tag] ciflow/inductor/164612 -> ciflow/inductor/164612 2025-10-10T00:37:13.3427201Z * [new tag] ciflow/inductor/164613 -> ciflow/inductor/164613 2025-10-10T00:37:13.3427392Z * [new tag] ciflow/inductor/164614 -> ciflow/inductor/164614 2025-10-10T00:37:13.3427515Z * [new tag] ciflow/inductor/164623 -> ciflow/inductor/164623 2025-10-10T00:37:13.3427639Z * [new tag] ciflow/inductor/164626 -> ciflow/inductor/164626 2025-10-10T00:37:13.3427773Z * [new tag] ciflow/inductor/164628 -> ciflow/inductor/164628 2025-10-10T00:37:13.3427897Z * [new tag] ciflow/inductor/164631 -> ciflow/inductor/164631 2025-10-10T00:37:13.3428030Z * [new tag] ciflow/inductor/164632 -> ciflow/inductor/164632 2025-10-10T00:37:13.3428155Z * [new tag] ciflow/inductor/164633 -> ciflow/inductor/164633 2025-10-10T00:37:13.3428280Z * [new tag] ciflow/inductor/164640 -> ciflow/inductor/164640 2025-10-10T00:37:13.3428411Z * [new tag] ciflow/inductor/164641 -> ciflow/inductor/164641 2025-10-10T00:37:13.3428540Z * [new tag] ciflow/inductor/164645 -> ciflow/inductor/164645 2025-10-10T00:37:13.3428672Z * [new tag] ciflow/inductor/164648 -> ciflow/inductor/164648 2025-10-10T00:37:13.3428800Z * [new tag] ciflow/inductor/164653 -> ciflow/inductor/164653 2025-10-10T00:37:13.3428936Z * [new tag] ciflow/inductor/164655 -> ciflow/inductor/164655 2025-10-10T00:37:13.3429063Z * [new tag] ciflow/inductor/164657 -> ciflow/inductor/164657 2025-10-10T00:37:13.3429226Z * [new tag] ciflow/inductor/164659 -> ciflow/inductor/164659 2025-10-10T00:37:13.3429358Z * [new tag] ciflow/inductor/164669 -> ciflow/inductor/164669 2025-10-10T00:37:13.3429481Z * [new tag] ciflow/inductor/164690 -> ciflow/inductor/164690 2025-10-10T00:37:13.3429612Z * [new tag] ciflow/inductor/164691 -> ciflow/inductor/164691 2025-10-10T00:37:13.3429739Z * [new tag] ciflow/inductor/164692 -> ciflow/inductor/164692 2025-10-10T00:37:13.3429861Z * [new tag] ciflow/inductor/164711 -> ciflow/inductor/164711 2025-10-10T00:37:13.3429991Z * [new tag] ciflow/inductor/164714 -> ciflow/inductor/164714 2025-10-10T00:37:13.3430113Z * [new tag] ciflow/inductor/164717 -> ciflow/inductor/164717 2025-10-10T00:37:13.3430242Z * [new tag] ciflow/inductor/164718 -> ciflow/inductor/164718 2025-10-10T00:37:13.3430365Z * [new tag] ciflow/inductor/164723 -> ciflow/inductor/164723 2025-10-10T00:37:13.3430498Z * [new tag] ciflow/inductor/164724 -> ciflow/inductor/164724 2025-10-10T00:37:13.3430620Z * [new tag] ciflow/inductor/164734 -> ciflow/inductor/164734 2025-10-10T00:37:13.3430742Z * [new tag] ciflow/inductor/164740 -> ciflow/inductor/164740 2025-10-10T00:37:13.3430871Z * [new tag] ciflow/inductor/164746 -> ciflow/inductor/164746 2025-10-10T00:37:13.3430997Z * [new tag] ciflow/inductor/164747 -> ciflow/inductor/164747 2025-10-10T00:37:13.3431127Z * [new tag] ciflow/inductor/164776 -> ciflow/inductor/164776 2025-10-10T00:37:13.3431251Z * [new tag] ciflow/inductor/164778 -> ciflow/inductor/164778 2025-10-10T00:37:13.3431374Z * [new tag] ciflow/inductor/164780 -> ciflow/inductor/164780 2025-10-10T00:37:13.3431729Z * [new tag] ciflow/inductor/164794 -> ciflow/inductor/164794 2025-10-10T00:37:13.3431866Z * [new tag] ciflow/inductor/164802 -> ciflow/inductor/164802 2025-10-10T00:37:13.3431999Z * [new tag] ciflow/inductor/164806 -> ciflow/inductor/164806 2025-10-10T00:37:13.3432121Z * [new tag] ciflow/inductor/164808 -> ciflow/inductor/164808 2025-10-10T00:37:13.3432252Z * [new tag] ciflow/inductor/164810 -> ciflow/inductor/164810 2025-10-10T00:37:13.3432466Z * [new tag] ciflow/inductor/164811 -> ciflow/inductor/164811 2025-10-10T00:37:13.3432591Z * [new tag] ciflow/inductor/164812 -> ciflow/inductor/164812 2025-10-10T00:37:13.3432723Z * [new tag] ciflow/inductor/164819 -> ciflow/inductor/164819 2025-10-10T00:37:13.3432846Z * [new tag] ciflow/inductor/164820 -> ciflow/inductor/164820 2025-10-10T00:37:13.3432976Z * [new tag] ciflow/inductor/164821 -> ciflow/inductor/164821 2025-10-10T00:37:13.3433103Z * [new tag] ciflow/inductor/164839 -> ciflow/inductor/164839 2025-10-10T00:37:13.3433228Z * [new tag] ciflow/inductor/164842 -> ciflow/inductor/164842 2025-10-10T00:37:13.3433359Z * [new tag] ciflow/inductor/164847 -> ciflow/inductor/164847 2025-10-10T00:37:13.3433484Z * [new tag] ciflow/inductor/164852 -> ciflow/inductor/164852 2025-10-10T00:37:13.3433636Z * [new tag] ciflow/inductor/164863 -> ciflow/inductor/164863 2025-10-10T00:37:13.3433758Z * [new tag] ciflow/inductor/164865 -> ciflow/inductor/164865 2025-10-10T00:37:13.3433887Z * [new tag] ciflow/inductor/164866 -> ciflow/inductor/164866 2025-10-10T00:37:13.3434010Z * [new tag] ciflow/inductor/164867 -> ciflow/inductor/164867 2025-10-10T00:37:13.3434133Z * [new tag] ciflow/inductor/164869 -> ciflow/inductor/164869 2025-10-10T00:37:13.3434336Z * [new tag] ciflow/inductor/164873 -> ciflow/inductor/164873 2025-10-10T00:37:13.3434461Z * [new tag] ciflow/inductor/164889 -> ciflow/inductor/164889 2025-10-10T00:37:13.3434595Z * [new tag] ciflow/inductor/164897 -> ciflow/inductor/164897 2025-10-10T00:37:13.3434722Z * [new tag] ciflow/inductor/164902 -> ciflow/inductor/164902 2025-10-10T00:37:13.3434854Z * [new tag] ciflow/inductor/164903 -> ciflow/inductor/164903 2025-10-10T00:37:13.3434978Z * [new tag] ciflow/inductor/164906 -> ciflow/inductor/164906 2025-10-10T00:37:13.3435100Z * [new tag] ciflow/inductor/164914 -> ciflow/inductor/164914 2025-10-10T00:37:13.3435354Z * [new tag] ciflow/inductor/164919 -> ciflow/inductor/164919 2025-10-10T00:37:13.3439040Z * [new tag] ciflow/inductor/164933 -> ciflow/inductor/164933 2025-10-10T00:37:13.3439195Z * [new tag] ciflow/inductor/164938 -> ciflow/inductor/164938 2025-10-10T00:37:13.3439404Z * [new tag] ciflow/inductor/164948 -> ciflow/inductor/164948 2025-10-10T00:37:13.3439555Z * [new tag] ciflow/inductor/164956 -> ciflow/inductor/164956 2025-10-10T00:37:13.3439694Z * [new tag] ciflow/inductor/164965 -> ciflow/inductor/164965 2025-10-10T00:37:13.3439833Z * [new tag] ciflow/inductor/164978 -> ciflow/inductor/164978 2025-10-10T00:37:13.3440025Z * [new tag] ciflow/inductor/164979 -> ciflow/inductor/164979 2025-10-10T00:37:13.3445830Z * [new tag] ciflow/inductor/164980 -> ciflow/inductor/164980 2025-10-10T00:37:13.3450880Z * [new tag] ciflow/inductor/164984 -> ciflow/inductor/164984 2025-10-10T00:37:13.3455768Z * [new tag] ciflow/inductor/164989 -> ciflow/inductor/164989 2025-10-10T00:37:13.3460245Z * [new tag] ciflow/inductor/164991 -> ciflow/inductor/164991 2025-10-10T00:37:13.3460410Z * [new tag] ciflow/inductor/164992 -> ciflow/inductor/164992 2025-10-10T00:37:13.3460565Z * [new tag] ciflow/inductor/164994 -> ciflow/inductor/164994 2025-10-10T00:37:13.3460698Z * [new tag] ciflow/inductor/164999 -> ciflow/inductor/164999 2025-10-10T00:37:13.3460969Z * [new tag] ciflow/inductor/165001 -> ciflow/inductor/165001 2025-10-10T00:37:13.3461107Z * [new tag] ciflow/inductor/165005 -> ciflow/inductor/165005 2025-10-10T00:37:13.3461233Z * [new tag] ciflow/inductor/165006 -> ciflow/inductor/165006 2025-10-10T00:37:13.3461364Z * [new tag] ciflow/inductor/165010 -> ciflow/inductor/165010 2025-10-10T00:37:13.3461486Z * [new tag] ciflow/inductor/165012 -> ciflow/inductor/165012 2025-10-10T00:37:13.3461647Z * [new tag] ciflow/inductor/165017 -> ciflow/inductor/165017 2025-10-10T00:37:13.3461772Z * [new tag] ciflow/inductor/165018 -> ciflow/inductor/165018 2025-10-10T00:37:13.3461905Z * [new tag] ciflow/inductor/165024 -> ciflow/inductor/165024 2025-10-10T00:37:13.3462033Z * [new tag] ciflow/inductor/165029 -> ciflow/inductor/165029 2025-10-10T00:37:13.3462161Z * [new tag] ciflow/inductor/165030 -> ciflow/inductor/165030 2025-10-10T00:37:13.3462293Z * [new tag] ciflow/inductor/165031 -> ciflow/inductor/165031 2025-10-10T00:37:13.3462416Z * [new tag] ciflow/inductor/165033 -> ciflow/inductor/165033 2025-10-10T00:37:13.3462545Z * [new tag] ciflow/inductor/165036 -> ciflow/inductor/165036 2025-10-10T00:37:13.3462670Z * [new tag] ciflow/inductor/165037 -> ciflow/inductor/165037 2025-10-10T00:37:13.3462802Z * [new tag] ciflow/inductor/165039 -> ciflow/inductor/165039 2025-10-10T00:37:13.3462970Z * [new tag] ciflow/inductor/165047 -> ciflow/inductor/165047 2025-10-10T00:37:13.3463097Z * [new tag] ciflow/inductor/165059 -> ciflow/inductor/165059 2025-10-10T00:37:13.3463228Z * [new tag] ciflow/inductor/165063 -> ciflow/inductor/165063 2025-10-10T00:37:13.3463353Z * [new tag] ciflow/inductor/165064 -> ciflow/inductor/165064 2025-10-10T00:37:13.3463488Z * [new tag] ciflow/inductor/165066 -> ciflow/inductor/165066 2025-10-10T00:37:13.3463610Z * [new tag] ciflow/inductor/165074 -> ciflow/inductor/165074 2025-10-10T00:37:13.3463735Z * [new tag] ciflow/inductor/165076 -> ciflow/inductor/165076 2025-10-10T00:37:13.3463866Z * [new tag] ciflow/inductor/165091 -> ciflow/inductor/165091 2025-10-10T00:37:13.3463992Z * [new tag] ciflow/inductor/165092 -> ciflow/inductor/165092 2025-10-10T00:37:13.3464137Z * [new tag] ciflow/inductor/165106 -> ciflow/inductor/165106 2025-10-10T00:37:13.3464259Z * [new tag] ciflow/inductor/165107 -> ciflow/inductor/165107 2025-10-10T00:37:13.3464390Z * [new tag] ciflow/inductor/165112 -> ciflow/inductor/165112 2025-10-10T00:37:13.3464515Z * [new tag] ciflow/inductor/165113 -> ciflow/inductor/165113 2025-10-10T00:37:13.3464656Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-10-10T00:37:13.3464798Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-10-10T00:37:13.3464940Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-10-10T00:37:13.3465100Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-10-10T00:37:13.3465248Z * [new tag] ciflow/linux-aarch64/163952 -> ciflow/linux-aarch64/163952 2025-10-10T00:37:13.3465400Z * [new tag] ciflow/linux-aarch64/164965 -> ciflow/linux-aarch64/164965 2025-10-10T00:37:13.3465550Z * [new tag] ciflow/linux-aarch64/165010 -> ciflow/linux-aarch64/165010 2025-10-10T00:37:13.3465677Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-10-10T00:37:13.3465805Z * [new tag] ciflow/mps/157554 -> ciflow/mps/157554 2025-10-10T00:37:13.3465952Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-10-10T00:37:13.3466068Z * [new tag] ciflow/mps/162340 -> ciflow/mps/162340 2025-10-10T00:37:13.3466180Z * [new tag] ciflow/mps/164416 -> ciflow/mps/164416 2025-10-10T00:37:13.3466474Z * [new tag] ciflow/mps/164571 -> ciflow/mps/164571 2025-10-10T00:37:13.3466608Z * [new tag] ciflow/mps/164965 -> ciflow/mps/164965 2025-10-10T00:37:13.3466749Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-10-10T00:37:13.3466888Z * [new tag] ciflow/nightly/164747 -> ciflow/nightly/164747 2025-10-10T00:37:13.3467013Z * [new tag] ciflow/nightly/164901 -> ciflow/nightly/164901 2025-10-10T00:37:13.3467167Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-10-10T00:37:13.3467319Z * [new tag] ciflow/op-benchmark/163767 -> ciflow/op-benchmark/163767 2025-10-10T00:37:13.3467467Z * [new tag] ciflow/op-benchmark/164583 -> ciflow/op-benchmark/164583 2025-10-10T00:37:13.3467618Z * [new tag] ciflow/op-benchmark/164747 -> ciflow/op-benchmark/164747 2025-10-10T00:37:13.3467814Z * [new tag] ciflow/periodic-rocm-mi300/162478 -> ciflow/periodic-rocm-mi300/162478 2025-10-10T00:37:13.3468002Z * [new tag] ciflow/periodic-rocm-mi300/163767 -> ciflow/periodic-rocm-mi300/163767 2025-10-10T00:37:13.3468237Z * [new tag] ciflow/periodic-rocm-mi300/164618 -> ciflow/periodic-rocm-mi300/164618 2025-10-10T00:37:13.3468408Z * [new tag] ciflow/periodic-rocm-mi300/164747 -> ciflow/periodic-rocm-mi300/164747 2025-10-10T00:37:13.3474227Z * [new tag] ciflow/periodic-rocm-mi300/165011 -> ciflow/periodic-rocm-mi300/165011 2025-10-10T00:37:13.3479293Z * [new tag] ciflow/periodic-rocm-mi300/165080 -> ciflow/periodic-rocm-mi300/165080 2025-10-10T00:37:13.3479488Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-10-10T00:37:13.3479823Z * [new tag] ciflow/periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> ciflow/periodic/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T00:37:13.3480150Z * [new tag] ciflow/periodic/0ea59c3c55dab37a6edefcc7002bb1428afd6456 -> ciflow/periodic/0ea59c3c55dab37a6edefcc7002bb1428afd6456 2025-10-10T00:37:13.3480322Z * [new tag] ciflow/periodic/156491 -> ciflow/periodic/156491 2025-10-10T00:37:13.3480464Z * [new tag] ciflow/periodic/162990 -> ciflow/periodic/162990 2025-10-10T00:37:13.3480597Z * [new tag] ciflow/periodic/163667 -> ciflow/periodic/163667 2025-10-10T00:37:13.3480730Z * [new tag] ciflow/periodic/163767 -> ciflow/periodic/163767 2025-10-10T00:37:13.3480869Z * [new tag] ciflow/periodic/164747 -> ciflow/periodic/164747 2025-10-10T00:37:13.3480995Z * [new tag] ciflow/periodic/164769 -> ciflow/periodic/164769 2025-10-10T00:37:13.3481127Z * [new tag] ciflow/periodic/165011 -> ciflow/periodic/165011 2025-10-10T00:37:13.3481439Z * [new tag] ciflow/periodic/2a6cdba6e5f74c2294fecc2d1344537522efbaab -> ciflow/periodic/2a6cdba6e5f74c2294fecc2d1344537522efbaab 2025-10-10T00:37:13.3481583Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-10-10T00:37:13.3481729Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-10-10T00:37:13.3481858Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-10-10T00:37:13.3481996Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-10-10T00:37:13.3482310Z * [new tag] ciflow/periodic/4bcc05777e780e834d44a2d06dd5321daec316f0 -> ciflow/periodic/4bcc05777e780e834d44a2d06dd5321daec316f0 2025-10-10T00:37:13.3482805Z * [new tag] ciflow/periodic/73adac05d13babb75410c3e033fdce57aa16881a -> ciflow/periodic/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T00:37:13.3482957Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-10-10T00:37:13.3483261Z * [new tag] ciflow/periodic/ac08556f674259ff5b117964e300124e8a92d45b -> ciflow/periodic/ac08556f674259ff5b117964e300124e8a92d45b 2025-10-10T00:37:13.3483423Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-10-10T00:37:13.3483579Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-10-10T00:37:13.3483736Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-10-10T00:37:13.3483891Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-10-10T00:37:13.3484062Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-10-10T00:37:13.3484217Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-10-10T00:37:13.3484413Z * [new tag] ciflow/quantization-periodic/163767 -> ciflow/quantization-periodic/163767 2025-10-10T00:37:13.3484624Z * [new tag] ciflow/quantization-periodic/164747 -> ciflow/quantization-periodic/164747 2025-10-10T00:37:13.3484752Z * [new tag] ciflow/riscv64/163767 -> ciflow/riscv64/163767 2025-10-10T00:37:13.3484926Z * [new tag] ciflow/riscv64/164747 -> ciflow/riscv64/164747 2025-10-10T00:37:13.3485066Z * [new tag] ciflow/rocm-mi300/161280 -> ciflow/rocm-mi300/161280 2025-10-10T00:37:13.3485200Z * [new tag] ciflow/rocm-mi300/162478 -> ciflow/rocm-mi300/162478 2025-10-10T00:37:13.3485326Z * [new tag] ciflow/rocm-mi300/163767 -> ciflow/rocm-mi300/163767 2025-10-10T00:37:13.3485455Z * [new tag] ciflow/rocm-mi300/163955 -> ciflow/rocm-mi300/163955 2025-10-10T00:37:13.3485587Z * [new tag] ciflow/rocm-mi300/164618 -> ciflow/rocm-mi300/164618 2025-10-10T00:37:13.3486114Z * [new tag] ciflow/rocm-mi300/164747 -> ciflow/rocm-mi300/164747 2025-10-10T00:37:13.3491073Z * [new tag] ciflow/rocm-mi300/164927 -> ciflow/rocm-mi300/164927 2025-10-10T00:37:13.3496102Z * [new tag] ciflow/rocm-mi300/164930 -> ciflow/rocm-mi300/164930 2025-10-10T00:37:13.3498054Z * [new tag] ciflow/rocm-mi300/165026 -> ciflow/rocm-mi300/165026 2025-10-10T00:37:13.3498319Z * [new tag] ciflow/rocm-mi300/165080 -> ciflow/rocm-mi300/165080 2025-10-10T00:37:13.3502457Z * [new tag] ciflow/rocm-mi355/163767 -> ciflow/rocm-mi355/163767 2025-10-10T00:37:13.3502949Z * [new tag] ciflow/rocm-mi355/164747 -> ciflow/rocm-mi355/164747 2025-10-10T00:37:13.3503131Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-10-10T00:37:13.3503270Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-10-10T00:37:13.3503392Z * [new tag] ciflow/rocm/156592 -> ciflow/rocm/156592 2025-10-10T00:37:13.3503513Z * [new tag] ciflow/rocm/161280 -> ciflow/rocm/161280 2025-10-10T00:37:13.3503647Z * [new tag] ciflow/rocm/163767 -> ciflow/rocm/163767 2025-10-10T00:37:13.3503773Z * [new tag] ciflow/rocm/163955 -> ciflow/rocm/163955 2025-10-10T00:37:13.3503910Z * [new tag] ciflow/rocm/163965 -> ciflow/rocm/163965 2025-10-10T00:37:13.3504024Z * [new tag] ciflow/rocm/164656 -> ciflow/rocm/164656 2025-10-10T00:37:13.3504148Z * [new tag] ciflow/rocm/164747 -> ciflow/rocm/164747 2025-10-10T00:37:13.3504404Z * [new tag] ciflow/rocm/164769 -> ciflow/rocm/164769 2025-10-10T00:37:13.3504524Z * [new tag] ciflow/rocm/164927 -> ciflow/rocm/164927 2025-10-10T00:37:13.3504649Z * [new tag] ciflow/rocm/164930 -> ciflow/rocm/164930 2025-10-10T00:37:13.3504766Z * [new tag] ciflow/rocm/165026 -> ciflow/rocm/165026 2025-10-10T00:37:13.3504890Z * [new tag] ciflow/rocm/165103 -> ciflow/rocm/165103 2025-10-10T00:37:13.3505009Z * [new tag] ciflow/s390/164747 -> ciflow/s390/164747 2025-10-10T00:37:13.3505134Z * [new tag] ciflow/s390/164917 -> ciflow/s390/164917 2025-10-10T00:37:13.3505264Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-10-10T00:37:13.3505396Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-10-10T00:37:13.3505751Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-10-10T00:37:13.3505898Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-10-10T00:37:13.3506020Z * [new tag] ciflow/slow/163767 -> ciflow/slow/163767 2025-10-10T00:37:13.3506136Z * [new tag] ciflow/slow/164747 -> ciflow/slow/164747 2025-10-10T00:37:13.3506251Z * [new tag] ciflow/slow/164769 -> ciflow/slow/164769 2025-10-10T00:37:13.3506571Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-10-10T00:37:13.3506695Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-10-10T00:37:13.3506826Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-10-10T00:37:13.3506946Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-10-10T00:37:13.3507074Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-10-10T00:37:13.3507193Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-10-10T00:37:13.3507313Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-10-10T00:37:13.3507442Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-10-10T00:37:13.3507802Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-10-10T00:37:13.3507956Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-10-10T00:37:13.3508073Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-10-10T00:37:13.3508294Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-10-10T00:37:13.3508420Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-10-10T00:37:13.3508540Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-10-10T00:37:13.3508872Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-10-10T00:37:13.3509020Z * [new tag] ciflow/torchbench/164747 -> ciflow/torchbench/164747 2025-10-10T00:37:13.3509144Z * [new tag] ciflow/trunk/113258 -> ciflow/trunk/113258 2025-10-10T00:37:13.3509257Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-10-10T00:37:13.3509374Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-10-10T00:37:13.3509501Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-10-10T00:37:13.3509614Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-10-10T00:37:13.3509735Z * [new tag] ciflow/trunk/149003 -> ciflow/trunk/149003 2025-10-10T00:37:13.3509896Z * [new tag] ciflow/trunk/149536 -> ciflow/trunk/149536 2025-10-10T00:37:13.3510050Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-10-10T00:37:13.3510168Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-10-10T00:37:13.3510339Z * [new tag] ciflow/trunk/154279 -> ciflow/trunk/154279 2025-10-10T00:37:13.3510461Z * [new tag] ciflow/trunk/154983 -> ciflow/trunk/154983 2025-10-10T00:37:13.3510581Z * [new tag] ciflow/trunk/156418 -> ciflow/trunk/156418 2025-10-10T00:37:13.3510703Z * [new tag] ciflow/trunk/156592 -> ciflow/trunk/156592 2025-10-10T00:37:13.3510816Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-10-10T00:37:13.3510941Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-10-10T00:37:13.3511059Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-10-10T00:37:13.3516140Z * [new tag] ciflow/trunk/159104 -> ciflow/trunk/159104 2025-10-10T00:37:13.3521038Z * [new tag] ciflow/trunk/160266 -> ciflow/trunk/160266 2025-10-10T00:37:13.3523383Z * [new tag] ciflow/trunk/160328 -> ciflow/trunk/160328 2025-10-10T00:37:13.3526235Z * [new tag] ciflow/trunk/160329 -> ciflow/trunk/160329 2025-10-10T00:37:13.3526565Z * [new tag] ciflow/trunk/160539 -> ciflow/trunk/160539 2025-10-10T00:37:13.3526858Z * [new tag] ciflow/trunk/160610 -> ciflow/trunk/160610 2025-10-10T00:37:13.3531988Z * [new tag] ciflow/trunk/160843 -> ciflow/trunk/160843 2025-10-10T00:37:13.3536402Z * [new tag] ciflow/trunk/161035 -> ciflow/trunk/161035 2025-10-10T00:37:13.3538746Z * [new tag] ciflow/trunk/161320 -> ciflow/trunk/161320 2025-10-10T00:37:13.3539018Z * [new tag] ciflow/trunk/162031 -> ciflow/trunk/162031 2025-10-10T00:37:13.3542443Z * [new tag] ciflow/trunk/162066 -> ciflow/trunk/162066 2025-10-10T00:37:13.3542748Z * [new tag] ciflow/trunk/162203 -> ciflow/trunk/162203 2025-10-10T00:37:13.3542913Z * [new tag] ciflow/trunk/162340 -> ciflow/trunk/162340 2025-10-10T00:37:13.3543032Z * [new tag] ciflow/trunk/162542 -> ciflow/trunk/162542 2025-10-10T00:37:13.3543159Z * [new tag] ciflow/trunk/162899 -> ciflow/trunk/162899 2025-10-10T00:37:13.3543270Z * [new tag] ciflow/trunk/163034 -> ciflow/trunk/163034 2025-10-10T00:37:13.3543389Z * [new tag] ciflow/trunk/163332 -> ciflow/trunk/163332 2025-10-10T00:37:13.3543501Z * [new tag] ciflow/trunk/163446 -> ciflow/trunk/163446 2025-10-10T00:37:13.3543629Z * [new tag] ciflow/trunk/163490 -> ciflow/trunk/163490 2025-10-10T00:37:13.3543742Z * [new tag] ciflow/trunk/163527 -> ciflow/trunk/163527 2025-10-10T00:37:13.3543861Z * [new tag] ciflow/trunk/163533 -> ciflow/trunk/163533 2025-10-10T00:37:13.3543972Z * [new tag] ciflow/trunk/163671 -> ciflow/trunk/163671 2025-10-10T00:37:13.3544086Z * [new tag] ciflow/trunk/163767 -> ciflow/trunk/163767 2025-10-10T00:37:13.3549530Z * [new tag] ciflow/trunk/163846 -> ciflow/trunk/163846 2025-10-10T00:37:13.3554532Z * [new tag] ciflow/trunk/163899 -> ciflow/trunk/163899 2025-10-10T00:37:13.3556682Z * [new tag] ciflow/trunk/163955 -> ciflow/trunk/163955 2025-10-10T00:37:13.3556935Z * [new tag] ciflow/trunk/163976 -> ciflow/trunk/163976 2025-10-10T00:37:13.3562512Z * [new tag] ciflow/trunk/164040 -> ciflow/trunk/164040 2025-10-10T00:37:13.3568094Z * [new tag] ciflow/trunk/164130 -> ciflow/trunk/164130 2025-10-10T00:37:13.3570122Z * [new tag] ciflow/trunk/164144 -> ciflow/trunk/164144 2025-10-10T00:37:13.3570350Z * [new tag] ciflow/trunk/164202 -> ciflow/trunk/164202 2025-10-10T00:37:13.3575872Z * [new tag] ciflow/trunk/164318 -> ciflow/trunk/164318 2025-10-10T00:37:13.3577690Z * [new tag] ciflow/trunk/164414 -> ciflow/trunk/164414 2025-10-10T00:37:13.3577994Z * [new tag] ciflow/trunk/164416 -> ciflow/trunk/164416 2025-10-10T00:37:13.3578130Z * [new tag] ciflow/trunk/164437 -> ciflow/trunk/164437 2025-10-10T00:37:13.3578321Z * [new tag] ciflow/trunk/164467 -> ciflow/trunk/164467 2025-10-10T00:37:13.3578448Z * [new tag] ciflow/trunk/164500 -> ciflow/trunk/164500 2025-10-10T00:37:13.3578578Z * [new tag] ciflow/trunk/164510 -> ciflow/trunk/164510 2025-10-10T00:37:13.3578727Z * [new tag] ciflow/trunk/164519 -> ciflow/trunk/164519 2025-10-10T00:37:13.3578899Z * [new tag] ciflow/trunk/164542 -> ciflow/trunk/164542 2025-10-10T00:37:13.3579023Z * [new tag] ciflow/trunk/164560 -> ciflow/trunk/164560 2025-10-10T00:37:13.3579137Z * [new tag] ciflow/trunk/164566 -> ciflow/trunk/164566 2025-10-10T00:37:13.3579570Z * [new tag] ciflow/trunk/164623 -> ciflow/trunk/164623 2025-10-10T00:37:13.3579823Z * [new tag] ciflow/trunk/164628 -> ciflow/trunk/164628 2025-10-10T00:37:13.3579962Z * [new tag] ciflow/trunk/164641 -> ciflow/trunk/164641 2025-10-10T00:37:13.3580079Z * [new tag] ciflow/trunk/164643 -> ciflow/trunk/164643 2025-10-10T00:37:13.3580310Z * [new tag] ciflow/trunk/164645 -> ciflow/trunk/164645 2025-10-10T00:37:13.3580431Z * [new tag] ciflow/trunk/164653 -> ciflow/trunk/164653 2025-10-10T00:37:13.3580542Z * [new tag] ciflow/trunk/164655 -> ciflow/trunk/164655 2025-10-10T00:37:13.3580686Z * [new tag] ciflow/trunk/164691 -> ciflow/trunk/164691 2025-10-10T00:37:13.3580794Z * [new tag] ciflow/trunk/164692 -> ciflow/trunk/164692 2025-10-10T00:37:13.3580912Z * [new tag] ciflow/trunk/164705 -> ciflow/trunk/164705 2025-10-10T00:37:13.3581031Z * [new tag] ciflow/trunk/164746 -> ciflow/trunk/164746 2025-10-10T00:37:13.3581140Z * [new tag] ciflow/trunk/164747 -> ciflow/trunk/164747 2025-10-10T00:37:13.3581258Z * [new tag] ciflow/trunk/164790 -> ciflow/trunk/164790 2025-10-10T00:37:13.3581364Z * [new tag] ciflow/trunk/164808 -> ciflow/trunk/164808 2025-10-10T00:37:13.3581487Z * [new tag] ciflow/trunk/164812 -> ciflow/trunk/164812 2025-10-10T00:37:13.3581595Z * [new tag] ciflow/trunk/164836 -> ciflow/trunk/164836 2025-10-10T00:37:13.3581704Z * [new tag] ciflow/trunk/164842 -> ciflow/trunk/164842 2025-10-10T00:37:13.3581822Z * [new tag] ciflow/trunk/164882 -> ciflow/trunk/164882 2025-10-10T00:37:13.3581935Z * [new tag] ciflow/trunk/164889 -> ciflow/trunk/164889 2025-10-10T00:37:13.3582055Z * [new tag] ciflow/trunk/164894 -> ciflow/trunk/164894 2025-10-10T00:37:13.3582163Z * [new tag] ciflow/trunk/164930 -> ciflow/trunk/164930 2025-10-10T00:37:13.3582271Z * [new tag] ciflow/trunk/164953 -> ciflow/trunk/164953 2025-10-10T00:37:13.3582389Z * [new tag] ciflow/trunk/164976 -> ciflow/trunk/164976 2025-10-10T00:37:13.3582542Z * [new tag] ciflow/trunk/164999 -> ciflow/trunk/164999 2025-10-10T00:37:13.3582657Z * [new tag] ciflow/trunk/165000 -> ciflow/trunk/165000 2025-10-10T00:37:13.3582764Z * [new tag] ciflow/trunk/165017 -> ciflow/trunk/165017 2025-10-10T00:37:13.3582870Z * [new tag] ciflow/trunk/165018 -> ciflow/trunk/165018 2025-10-10T00:37:13.3582990Z * [new tag] ciflow/trunk/165024 -> ciflow/trunk/165024 2025-10-10T00:37:13.3583108Z * [new tag] ciflow/trunk/165033 -> ciflow/trunk/165033 2025-10-10T00:37:13.3583231Z * [new tag] ciflow/trunk/165047 -> ciflow/trunk/165047 2025-10-10T00:37:13.3583344Z * [new tag] ciflow/trunk/165057 -> ciflow/trunk/165057 2025-10-10T00:37:13.3583462Z * [new tag] ciflow/trunk/165060 -> ciflow/trunk/165060 2025-10-10T00:37:13.3583583Z * [new tag] ciflow/trunk/165065 -> ciflow/trunk/165065 2025-10-10T00:37:13.3583697Z * [new tag] ciflow/trunk/165066 -> ciflow/trunk/165066 2025-10-10T00:37:13.3583823Z * [new tag] ciflow/trunk/165090 -> ciflow/trunk/165090 2025-10-10T00:37:13.3583938Z * [new tag] ciflow/trunk/165094 -> ciflow/trunk/165094 2025-10-10T00:37:13.3584061Z * [new tag] ciflow/trunk/165113 -> ciflow/trunk/165113 2025-10-10T00:37:13.3584180Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-10-10T00:37:13.3584333Z * [new tag] ciflow/vllm/164628 -> ciflow/vllm/164628 2025-10-10T00:37:13.3584482Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-10-10T00:37:13.3584605Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-10-10T00:37:13.3584729Z * [new tag] ciflow/xpu/161485 -> ciflow/xpu/161485 2025-10-10T00:37:13.3584847Z * [new tag] ciflow/xpu/162454 -> ciflow/xpu/162454 2025-10-10T00:37:13.3584967Z * [new tag] ciflow/xpu/163332 -> ciflow/xpu/163332 2025-10-10T00:37:13.3585085Z * [new tag] cslpull75 -> cslpull75 2025-10-10T00:37:13.3585192Z * [new tag] cslpull76 -> cslpull76 2025-10-10T00:37:13.3585321Z * [new tag] cslpull77 -> cslpull77 2025-10-10T00:37:13.3585428Z * [new tag] cslpull78 -> cslpull78 2025-10-10T00:37:13.3585539Z * [new tag] cslpull79 -> cslpull79 2025-10-10T00:37:13.3585644Z * [new tag] cslpull80 -> cslpull80 2025-10-10T00:37:13.3585754Z * [new tag] cslpull81 -> cslpull81 2025-10-10T00:37:13.3585855Z * [new tag] cslpull82 -> cslpull82 2025-10-10T00:37:13.3585960Z * [new tag] cslpull83 -> cslpull83 2025-10-10T00:37:13.3586069Z * [new tag] cslpull84 -> cslpull84 2025-10-10T00:37:13.3586168Z * [new tag] cslpull85 -> cslpull85 2025-10-10T00:37:13.3586455Z * [new tag] cslpull86 -> cslpull86 2025-10-10T00:37:13.3586566Z * [new tag] cslpull87 -> cslpull87 2025-10-10T00:37:13.3586666Z * [new tag] cslpull88 -> cslpull88 2025-10-10T00:37:13.3586779Z * [new tag] cslpull89 -> cslpull89 2025-10-10T00:37:13.3586880Z * [new tag] cslpull90 -> cslpull90 2025-10-10T00:37:13.3586989Z * [new tag] cslpull91 -> cslpull91 2025-10-10T00:37:13.3587091Z * [new tag] cslpull92 -> cslpull92 2025-10-10T00:37:13.3587252Z * [new tag] flight_5 -> flight_5 2025-10-10T00:37:13.3587364Z * [new tag] flight_5.1 -> flight_5.1 2025-10-10T00:37:13.3587471Z * [new tag] flight_5.2 -> flight_5.2 2025-10-10T00:37:13.3587582Z * [new tag] flight_5.3 -> flight_5.3 2025-10-10T00:37:13.3587696Z * [new tag] forpull1 -> forpull1 2025-10-10T00:37:13.3587826Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-10-10T00:37:13.3587945Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-10-10T00:37:13.3588059Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-10-10T00:37:13.3588187Z * [new tag] nightly-binary -> nightly-binary 2025-10-10T00:37:13.3588312Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-10-10T00:37:13.3588434Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-10-10T00:37:13.3588687Z * [new tag] trunk/001e1d263746ae9d121d9c8cf55bc87f777d9dba -> trunk/001e1d263746ae9d121d9c8cf55bc87f777d9dba 2025-10-10T00:37:13.3588928Z * [new tag] trunk/005c3d449e4c655d2eb0d76949a8cd41ce88f979 -> trunk/005c3d449e4c655d2eb0d76949a8cd41ce88f979 2025-10-10T00:37:13.3589155Z * [new tag] trunk/00f0365b959323bab89dc0a5bd5d40589e78edc8 -> trunk/00f0365b959323bab89dc0a5bd5d40589e78edc8 2025-10-10T00:37:13.3589408Z * [new tag] trunk/01f3a43462da594b65a6c9e8b46c132cd360cea9 -> trunk/01f3a43462da594b65a6c9e8b46c132cd360cea9 2025-10-10T00:37:13.3589638Z * [new tag] trunk/0319556a35b01e8857f7bf75df9df3287e1e853a -> trunk/0319556a35b01e8857f7bf75df9df3287e1e853a 2025-10-10T00:37:13.3589863Z * [new tag] trunk/054268c9ebb3291c6fd442e4a1f6602a8ea43ab6 -> trunk/054268c9ebb3291c6fd442e4a1f6602a8ea43ab6 2025-10-10T00:37:13.3590096Z * [new tag] trunk/06d86e58d0309aa2c217256f88d1990a22ec6e4f -> trunk/06d86e58d0309aa2c217256f88d1990a22ec6e4f 2025-10-10T00:37:13.3590321Z * [new tag] trunk/078d475d3bb104823e70ce975c2ee0d4d2fb0952 -> trunk/078d475d3bb104823e70ce975c2ee0d4d2fb0952 2025-10-10T00:37:13.3590564Z * [new tag] trunk/086dec3235d463e751c12ce9eeeb2dfcc873e206 -> trunk/086dec3235d463e751c12ce9eeeb2dfcc873e206 2025-10-10T00:37:13.3590793Z * [new tag] trunk/0a3e4e894cbc0cc93568c5d016f3ad72650cf641 -> trunk/0a3e4e894cbc0cc93568c5d016f3ad72650cf641 2025-10-10T00:37:13.3591038Z * [new tag] trunk/0b01ff4de02035eb21c1bc6bf4b1b627bc1cefaa -> trunk/0b01ff4de02035eb21c1bc6bf4b1b627bc1cefaa 2025-10-10T00:37:13.3591271Z * [new tag] trunk/0b15f7ae059cf4fa3909bbb009d83c0253a6385a -> trunk/0b15f7ae059cf4fa3909bbb009d83c0253a6385a 2025-10-10T00:37:13.3591497Z * [new tag] trunk/0b4f2b46d9e14c1858dd3d0ca9b62e349ae316cf -> trunk/0b4f2b46d9e14c1858dd3d0ca9b62e349ae316cf 2025-10-10T00:37:13.3591733Z * [new tag] trunk/0b85236477fe8a0e32510bcc973b2f34ef981df2 -> trunk/0b85236477fe8a0e32510bcc973b2f34ef981df2 2025-10-10T00:37:13.3591962Z * [new tag] trunk/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 -> trunk/0d39ecb2ce8556e85343d8da0c87450192c2fdf8 2025-10-10T00:37:13.3592197Z * [new tag] trunk/0e5773b7fadef9e29b006af470b771fad55b5206 -> trunk/0e5773b7fadef9e29b006af470b771fad55b5206 2025-10-10T00:37:13.3592542Z * [new tag] trunk/0e9b3a772ab96e998ab85591d5b2a9c1d41bacb0 -> trunk/0e9b3a772ab96e998ab85591d5b2a9c1d41bacb0 2025-10-10T00:37:13.3592829Z * [new tag] trunk/0fbe3f19c7e88ee1720d2e1579e3fd2cafdaabf9 -> trunk/0fbe3f19c7e88ee1720d2e1579e3fd2cafdaabf9 2025-10-10T00:37:13.3593189Z * [new tag] trunk/0fd976b65c6daf3799a501d9202e4f50144446d1 -> trunk/0fd976b65c6daf3799a501d9202e4f50144446d1 2025-10-10T00:37:13.3593913Z * [new tag] trunk/1051c1de5c0c1d34bec94c4a3199ac7b23bb19e1 -> trunk/1051c1de5c0c1d34bec94c4a3199ac7b23bb19e1 2025-10-10T00:37:13.3594177Z * [new tag] trunk/115af42e9d57e89c26777be72822107cd7b39e07 -> trunk/115af42e9d57e89c26777be72822107cd7b39e07 2025-10-10T00:37:13.3594427Z * [new tag] trunk/11f5f656867089dac1fa1e64e34c9966578fbddd -> trunk/11f5f656867089dac1fa1e64e34c9966578fbddd 2025-10-10T00:37:13.3594684Z * [new tag] trunk/12d2ef557f6e127100267c31a31572d8ab5cc788 -> trunk/12d2ef557f6e127100267c31a31572d8ab5cc788 2025-10-10T00:37:13.3594944Z * [new tag] trunk/144378615a5a2b347e39c6376cba7d75f7a82926 -> trunk/144378615a5a2b347e39c6376cba7d75f7a82926 2025-10-10T00:37:13.3595187Z * [new tag] trunk/14791ea947349fb5fa7b7d6230cfd3924c36ba27 -> trunk/14791ea947349fb5fa7b7d6230cfd3924c36ba27 2025-10-10T00:37:13.3595436Z * [new tag] trunk/15800888b697bacd555399b3a0ca2e8d0827528e -> trunk/15800888b697bacd555399b3a0ca2e8d0827528e 2025-10-10T00:37:13.3595706Z * [new tag] trunk/15c8bdcc5e3a6dfd14e5c977438f772031e064ff -> trunk/15c8bdcc5e3a6dfd14e5c977438f772031e064ff 2025-10-10T00:37:13.3596089Z * [new tag] trunk/15d726005ddc5558c934c3edd5f815c2e504e501 -> trunk/15d726005ddc5558c934c3edd5f815c2e504e501 2025-10-10T00:37:13.3596411Z * [new tag] trunk/16f9bef642b07b3090a6e4a04517eff84d41a197 -> trunk/16f9bef642b07b3090a6e4a04517eff84d41a197 2025-10-10T00:37:13.3596809Z * [new tag] trunk/17c7170ca6e2efd5ead2b93bd12e226ff48f0669 -> trunk/17c7170ca6e2efd5ead2b93bd12e226ff48f0669 2025-10-10T00:37:13.3597097Z * [new tag] trunk/184817c7a81d5c01e107a84efeb269b063ddf5d6 -> trunk/184817c7a81d5c01e107a84efeb269b063ddf5d6 2025-10-10T00:37:13.3602599Z * [new tag] trunk/18940820006d2304460008575561e2e8e7fc59fc -> trunk/18940820006d2304460008575561e2e8e7fc59fc 2025-10-10T00:37:13.3605704Z * [new tag] trunk/18e18488e8c90e53cc113b1a5eddd9640ee80292 -> trunk/18e18488e8c90e53cc113b1a5eddd9640ee80292 2025-10-10T00:37:13.3605973Z * [new tag] trunk/1927783aa3ad676db6f4c34fc77ef3825a4e2ed5 -> trunk/1927783aa3ad676db6f4c34fc77ef3825a4e2ed5 2025-10-10T00:37:13.3606266Z * [new tag] trunk/19bf67be3286c0e2babe83af0d1593bae850362a -> trunk/19bf67be3286c0e2babe83af0d1593bae850362a 2025-10-10T00:37:13.3606493Z * [new tag] trunk/1bb68271b7ff1b582845384c6c7f7b1593ae1619 -> trunk/1bb68271b7ff1b582845384c6c7f7b1593ae1619 2025-10-10T00:37:13.3606728Z * [new tag] trunk/1d182dd81c3143697337e35d046fd02951dedb09 -> trunk/1d182dd81c3143697337e35d046fd02951dedb09 2025-10-10T00:37:13.3606966Z * [new tag] trunk/1e42fde45eff81845f269e8185f54a19f6d87c5b -> trunk/1e42fde45eff81845f269e8185f54a19f6d87c5b 2025-10-10T00:37:13.3607304Z * [new tag] trunk/1f73b96668bc6ae4c8e7ef5b630ff5f3c69ae005 -> trunk/1f73b96668bc6ae4c8e7ef5b630ff5f3c69ae005 2025-10-10T00:37:13.3611101Z * [new tag] trunk/1f8ee5da117952b03f0050a178d69f8e7189b0f8 -> trunk/1f8ee5da117952b03f0050a178d69f8e7189b0f8 2025-10-10T00:37:13.3613564Z * [new tag] trunk/1f9614cef8e0272c8e3bd99004d2978a6ecc5195 -> trunk/1f9614cef8e0272c8e3bd99004d2978a6ecc5195 2025-10-10T00:37:13.3613855Z * [new tag] trunk/1fb072ac2a33af93a77888dddbdd228b22a3f9c4 -> trunk/1fb072ac2a33af93a77888dddbdd228b22a3f9c4 2025-10-10T00:37:13.3614097Z * [new tag] trunk/1fc71d1b578badb1b3ba7cc2d5795f4f80463749 -> trunk/1fc71d1b578badb1b3ba7cc2d5795f4f80463749 2025-10-10T00:37:13.3614353Z * [new tag] trunk/20082d713666fa1eade588bebd523d86309bfa25 -> trunk/20082d713666fa1eade588bebd523d86309bfa25 2025-10-10T00:37:13.3614592Z * [new tag] trunk/2164b661219ab0a76aa018e955ba3d8e8f99c083 -> trunk/2164b661219ab0a76aa018e955ba3d8e8f99c083 2025-10-10T00:37:13.3614834Z * [new tag] trunk/228973df7f770505aafc6fc17b99f81ac58bdfe1 -> trunk/228973df7f770505aafc6fc17b99f81ac58bdfe1 2025-10-10T00:37:13.3615243Z * [new tag] trunk/22b1710252525d80d47ba95c762ccdbf577b2dc2 -> trunk/22b1710252525d80d47ba95c762ccdbf577b2dc2 2025-10-10T00:37:13.3615485Z * [new tag] trunk/22e219d9969ff3cee85bc5de32fa49d5a549a148 -> trunk/22e219d9969ff3cee85bc5de32fa49d5a549a148 2025-10-10T00:37:13.3615727Z * [new tag] trunk/235b995ce18de632ab816940319fcd66b46039b8 -> trunk/235b995ce18de632ab816940319fcd66b46039b8 2025-10-10T00:37:13.3615972Z * [new tag] trunk/23ab6a45e5c759fb4714905cb8c84ef74c70aa67 -> trunk/23ab6a45e5c759fb4714905cb8c84ef74c70aa67 2025-10-10T00:37:13.3616226Z * [new tag] trunk/24d69c57cbaa94cc828dbbdf83c889f5f244ae28 -> trunk/24d69c57cbaa94cc828dbbdf83c889f5f244ae28 2025-10-10T00:37:13.3616467Z * [new tag] trunk/263db92563f0ae71bf3e4fc265fbb48e79f9f23f -> trunk/263db92563f0ae71bf3e4fc265fbb48e79f9f23f 2025-10-10T00:37:13.3616721Z * [new tag] trunk/27234792add2ee9bedd84ca02dbf34f8f244bc5c -> trunk/27234792add2ee9bedd84ca02dbf34f8f244bc5c 2025-10-10T00:37:13.3616968Z * [new tag] trunk/27eb36debbe3fa2d43a2f893a5c46a6257a09460 -> trunk/27eb36debbe3fa2d43a2f893a5c46a6257a09460 2025-10-10T00:37:13.3617204Z * [new tag] trunk/2855a045b30dafad7a08d66e242be13770189c19 -> trunk/2855a045b30dafad7a08d66e242be13770189c19 2025-10-10T00:37:13.3617450Z * [new tag] trunk/2883b5ab773daf5861d43ff0b65be49a441ab3f9 -> trunk/2883b5ab773daf5861d43ff0b65be49a441ab3f9 2025-10-10T00:37:13.3617811Z * [new tag] trunk/29824067215f3ed9e4044ca0f31a71e9d95f237d -> trunk/29824067215f3ed9e4044ca0f31a71e9d95f237d 2025-10-10T00:37:13.3618059Z * [new tag] trunk/2a11ce2c787b2339ffb8941b849dd487d25b4121 -> trunk/2a11ce2c787b2339ffb8941b849dd487d25b4121 2025-10-10T00:37:13.3618299Z * [new tag] trunk/2a6cdba6e5f74c2294fecc2d1344537522efbaab -> trunk/2a6cdba6e5f74c2294fecc2d1344537522efbaab 2025-10-10T00:37:13.3618544Z * [new tag] trunk/2a760dc51e04d65845440cc09e7016cfc74f9132 -> trunk/2a760dc51e04d65845440cc09e7016cfc74f9132 2025-10-10T00:37:13.3618776Z * [new tag] trunk/2a7c48675010056f23d62b5c6ecb318782801723 -> trunk/2a7c48675010056f23d62b5c6ecb318782801723 2025-10-10T00:37:13.3619019Z * [new tag] trunk/2b58adc3bdcf9476e1cef49ad965b7d3c7b9ac24 -> trunk/2b58adc3bdcf9476e1cef49ad965b7d3c7b9ac24 2025-10-10T00:37:13.3619268Z * [new tag] trunk/2b9ff9953523a2e916234c9197d946f4cff976c7 -> trunk/2b9ff9953523a2e916234c9197d946f4cff976c7 2025-10-10T00:37:13.3619501Z * [new tag] trunk/2c2e1268b7aae8ed610d12f2d38d39f8d93888a3 -> trunk/2c2e1268b7aae8ed610d12f2d38d39f8d93888a3 2025-10-10T00:37:13.3619739Z * [new tag] trunk/2c5ed6e7c067573b093725cd15d13812d9647562 -> trunk/2c5ed6e7c067573b093725cd15d13812d9647562 2025-10-10T00:37:13.3619989Z * [new tag] trunk/2d50678dcc7ab2da13a9bca6af8f2333e8970344 -> trunk/2d50678dcc7ab2da13a9bca6af8f2333e8970344 2025-10-10T00:37:13.3620239Z * [new tag] trunk/2e027e874232fefe7b1c56ce8aeb26c0e6b97f15 -> trunk/2e027e874232fefe7b1c56ce8aeb26c0e6b97f15 2025-10-10T00:37:13.3620477Z * [new tag] trunk/2e1742dd63c2168fd9649dbba96a95abf1f57cae -> trunk/2e1742dd63c2168fd9649dbba96a95abf1f57cae 2025-10-10T00:37:13.3620739Z * [new tag] trunk/2fe37b5fde392535a3238f975c93dd202cd3e24b -> trunk/2fe37b5fde392535a3238f975c93dd202cd3e24b 2025-10-10T00:37:13.3620980Z * [new tag] trunk/3040a5d294bd30d3938d0043a5d93d6c23264827 -> trunk/3040a5d294bd30d3938d0043a5d93d6c23264827 2025-10-10T00:37:13.3621217Z * [new tag] trunk/321e6026925f6b6e8a36e3a8b7c0295cd7541911 -> trunk/321e6026925f6b6e8a36e3a8b7c0295cd7541911 2025-10-10T00:37:13.3621456Z * [new tag] trunk/322091d8d8542a0cbff524306029bef4d7338747 -> trunk/322091d8d8542a0cbff524306029bef4d7338747 2025-10-10T00:37:13.3621733Z * [new tag] trunk/3288fbf374128610928e27d03615ac0d46a6ce14 -> trunk/3288fbf374128610928e27d03615ac0d46a6ce14 2025-10-10T00:37:13.3622160Z * [new tag] trunk/331191ce4b29b5d7d3bb7f0e7454ca70c06fbd26 -> trunk/331191ce4b29b5d7d3bb7f0e7454ca70c06fbd26 2025-10-10T00:37:13.3622512Z * [new tag] trunk/33b17bc619b044a0050797987efb8890d43319df -> trunk/33b17bc619b044a0050797987efb8890d43319df 2025-10-10T00:37:13.3622871Z * [new tag] trunk/34042a9145fe28033e7edb08f1fcf90ed197f4ac -> trunk/34042a9145fe28033e7edb08f1fcf90ed197f4ac 2025-10-10T00:37:13.3623233Z * [new tag] trunk/344e6365a0068c2d2847fcec0c55dd53291d475e -> trunk/344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:37:13.3623697Z * [new tag] trunk/34ac9b61cbfcf17328ccb8b729509829447fdddd -> trunk/34ac9b61cbfcf17328ccb8b729509829447fdddd 2025-10-10T00:37:13.3624614Z * [new tag] trunk/35c4130fd1358c98e12301ffa0f1b2294e0c795f -> trunk/35c4130fd1358c98e12301ffa0f1b2294e0c795f 2025-10-10T00:37:13.3624987Z * [new tag] trunk/35f66b83f89a571d0c0abe16c66a23120b92bdaf -> trunk/35f66b83f89a571d0c0abe16c66a23120b92bdaf 2025-10-10T00:37:13.3625566Z * [new tag] trunk/361c5d362c4ea1950e05116899cfcf753c345ebd -> trunk/361c5d362c4ea1950e05116899cfcf753c345ebd 2025-10-10T00:37:13.3628746Z * [new tag] trunk/37c6087334cce3ad4bc9838ea2ef63aba89f2253 -> trunk/37c6087334cce3ad4bc9838ea2ef63aba89f2253 2025-10-10T00:37:13.3629358Z * [new tag] trunk/3912ba3e940b9354622fa09b2ada677cd10723d8 -> trunk/3912ba3e940b9354622fa09b2ada677cd10723d8 2025-10-10T00:37:13.3634338Z * [new tag] trunk/39189592fd688979e56063430ed5a038d999908f -> trunk/39189592fd688979e56063430ed5a038d999908f 2025-10-10T00:37:13.3636613Z * [new tag] trunk/3924f784ba81f87fe09988d6fc9620b57e4d9f72 -> trunk/3924f784ba81f87fe09988d6fc9620b57e4d9f72 2025-10-10T00:37:13.3637048Z * [new tag] trunk/39b31a6bfde6e046383ae2b06fe0b68df5cdbdd2 -> trunk/39b31a6bfde6e046383ae2b06fe0b68df5cdbdd2 2025-10-10T00:37:13.3642586Z * [new tag] trunk/39c340ec9e2ee3011f1d260f581b5a95f3c99039 -> trunk/39c340ec9e2ee3011f1d260f581b5a95f3c99039 2025-10-10T00:37:13.3644931Z * [new tag] trunk/39d0c06ed0d7bc634d7f1a4e84b69f66d1ea0798 -> trunk/39d0c06ed0d7bc634d7f1a4e84b69f66d1ea0798 2025-10-10T00:37:13.3650497Z * [new tag] trunk/3c0577bd15778c96cecf0e7a5e5958d7fcab64f0 -> trunk/3c0577bd15778c96cecf0e7a5e5958d7fcab64f0 2025-10-10T00:37:13.3653077Z * [new tag] trunk/3c59351c6ea2fc29d346903e28e95c5f4d0ccdbb -> trunk/3c59351c6ea2fc29d346903e28e95c5f4d0ccdbb 2025-10-10T00:37:13.3659328Z * [new tag] trunk/3c5ca685d6f5b6f3971c0cd20a054aa355610419 -> trunk/3c5ca685d6f5b6f3971c0cd20a054aa355610419 2025-10-10T00:37:13.3662348Z * [new tag] trunk/3ca09d65f1bdf83142dc9fe47976227ae4a88e7b -> trunk/3ca09d65f1bdf83142dc9fe47976227ae4a88e7b 2025-10-10T00:37:13.3662651Z * [new tag] trunk/3cc8af2d67f42bf2a933796290446c5ab8978aac -> trunk/3cc8af2d67f42bf2a933796290446c5ab8978aac 2025-10-10T00:37:13.3662924Z * [new tag] trunk/3d1fa40ae1fee18ddf3dca89229e3ae828589e0c -> trunk/3d1fa40ae1fee18ddf3dca89229e3ae828589e0c 2025-10-10T00:37:13.3663172Z * [new tag] trunk/3d9d41c80168bcd3c569345a96682c42a5eba36a -> trunk/3d9d41c80168bcd3c569345a96682c42a5eba36a 2025-10-10T00:37:13.3663423Z * [new tag] trunk/3db21643417a04f6f2707a783ac32a538a98d53d -> trunk/3db21643417a04f6f2707a783ac32a538a98d53d 2025-10-10T00:37:13.3663693Z * [new tag] trunk/3ddf2018d0b7b4def0553dc092d928ef831a19c3 -> trunk/3ddf2018d0b7b4def0553dc092d928ef831a19c3 2025-10-10T00:37:13.3663957Z * [new tag] trunk/3e03deab6f3c268c85c8efd9546e28cdda0fa4cc -> trunk/3e03deab6f3c268c85c8efd9546e28cdda0fa4cc 2025-10-10T00:37:13.3664211Z * [new tag] trunk/3e0826c9d792ae87373dc0ff5d46c260020de29f -> trunk/3e0826c9d792ae87373dc0ff5d46c260020de29f 2025-10-10T00:37:13.3664646Z * [new tag] trunk/409aece3f9436a2f740f1b97f1243f738f6bbbf6 -> trunk/409aece3f9436a2f740f1b97f1243f738f6bbbf6 2025-10-10T00:37:13.3664906Z * [new tag] trunk/40b25578e4ecb7ef1c38201b3ce0014eb57c53eb -> trunk/40b25578e4ecb7ef1c38201b3ce0014eb57c53eb 2025-10-10T00:37:13.3665162Z * [new tag] trunk/412c6d28ec3869ef8ba962b290d755251e7cc3c1 -> trunk/412c6d28ec3869ef8ba962b290d755251e7cc3c1 2025-10-10T00:37:13.3665415Z * [new tag] trunk/415e641572473479fc9d9eaea12762e1a223a9e0 -> trunk/415e641572473479fc9d9eaea12762e1a223a9e0 2025-10-10T00:37:13.3665669Z * [new tag] trunk/41808b2ba9a61ab2f4c7af394c1668d09a4a0331 -> trunk/41808b2ba9a61ab2f4c7af394c1668d09a4a0331 2025-10-10T00:37:13.3665913Z * [new tag] trunk/4308b8a28fa332d23ad6d25a472559b354619131 -> trunk/4308b8a28fa332d23ad6d25a472559b354619131 2025-10-10T00:37:13.3666175Z * [new tag] trunk/43848b71d9af0223eafdd1755bf7444aafe9e993 -> trunk/43848b71d9af0223eafdd1755bf7444aafe9e993 2025-10-10T00:37:13.3666589Z * [new tag] trunk/43fc859625f9c0a794307b3ef30c26ab3fc2bfec -> trunk/43fc859625f9c0a794307b3ef30c26ab3fc2bfec 2025-10-10T00:37:13.3666843Z * [new tag] trunk/4412026949b562f940d4c24162de19d299725b62 -> trunk/4412026949b562f940d4c24162de19d299725b62 2025-10-10T00:37:13.3667086Z * [new tag] trunk/44a5d419935a77b3308f247279a457e6d0b9a292 -> trunk/44a5d419935a77b3308f247279a457e6d0b9a292 2025-10-10T00:37:13.3667405Z * [new tag] trunk/4661200125ba9c87aa7d54a55e585403b5ce5040 -> trunk/4661200125ba9c87aa7d54a55e585403b5ce5040 2025-10-10T00:37:13.3667628Z * [new tag] trunk/4691fe60700ac51a878775fd23a8f7c4548c6757 -> trunk/4691fe60700ac51a878775fd23a8f7c4548c6757 2025-10-10T00:37:13.3667845Z * [new tag] trunk/4725871a815fb880e89135a493c8c94ab9bbfece -> trunk/4725871a815fb880e89135a493c8c94ab9bbfece 2025-10-10T00:37:13.3668076Z * [new tag] trunk/47956196d99166fe9083beb2a52fd2e6c90b2011 -> trunk/47956196d99166fe9083beb2a52fd2e6c90b2011 2025-10-10T00:37:13.3668297Z * [new tag] trunk/483f4e0db91166128ad8922d86dc7222338d4ecc -> trunk/483f4e0db91166128ad8922d86dc7222338d4ecc 2025-10-10T00:37:13.3668571Z * [new tag] trunk/48b54b45d62af7ecafccc5afede04474cb236f1a -> trunk/48b54b45d62af7ecafccc5afede04474cb236f1a 2025-10-10T00:37:13.3668794Z * [new tag] trunk/49f7d8d19d24f616b11ef050535a211245aed649 -> trunk/49f7d8d19d24f616b11ef050535a211245aed649 2025-10-10T00:37:13.3669029Z * [new tag] trunk/4a0df39f814afad087e8b29dd2914a8b54567694 -> trunk/4a0df39f814afad087e8b29dd2914a8b54567694 2025-10-10T00:37:13.3669264Z * [new tag] trunk/4a6abba0d9fb3dc0f29b5efe527e26b2962caec1 -> trunk/4a6abba0d9fb3dc0f29b5efe527e26b2962caec1 2025-10-10T00:37:13.3669510Z * [new tag] trunk/4ab847bbc7ba09f29a4e81494e8a752dcb411117 -> trunk/4ab847bbc7ba09f29a4e81494e8a752dcb411117 2025-10-10T00:37:13.3669738Z * [new tag] trunk/4bcc05777e780e834d44a2d06dd5321daec316f0 -> trunk/4bcc05777e780e834d44a2d06dd5321daec316f0 2025-10-10T00:37:13.3669976Z * [new tag] trunk/4bd1505f849e701a8e54f9d185c23f13e7324498 -> trunk/4bd1505f849e701a8e54f9d185c23f13e7324498 2025-10-10T00:37:13.3670215Z * [new tag] trunk/4c0fec3e4dac35b9e9dec2beacfb5967906a4701 -> trunk/4c0fec3e4dac35b9e9dec2beacfb5967906a4701 2025-10-10T00:37:13.3670445Z * [new tag] trunk/4c3c0ef2f10415f5d5b13f1c842f91bb90ee91d3 -> trunk/4c3c0ef2f10415f5d5b13f1c842f91bb90ee91d3 2025-10-10T00:37:13.3670676Z * [new tag] trunk/4d7f9f3aed68729380730ed46e29ff2052f05b73 -> trunk/4d7f9f3aed68729380730ed46e29ff2052f05b73 2025-10-10T00:37:13.3670899Z * [new tag] trunk/50e077beaaf71798f870552f3849e4a52c784df5 -> trunk/50e077beaaf71798f870552f3849e4a52c784df5 2025-10-10T00:37:13.3671162Z * [new tag] trunk/5103ecc5d8f0cc90e686763652e2d84c22d83ca9 -> trunk/5103ecc5d8f0cc90e686763652e2d84c22d83ca9 2025-10-10T00:37:13.3671385Z * [new tag] trunk/5178d0a480f8f4e21da3757de455c8215b249ec5 -> trunk/5178d0a480f8f4e21da3757de455c8215b249ec5 2025-10-10T00:37:13.3671613Z * [new tag] trunk/5209c8ce0704f34ba4bd2a58c19877fbf6cf0392 -> trunk/5209c8ce0704f34ba4bd2a58c19877fbf6cf0392 2025-10-10T00:37:13.3671833Z * [new tag] trunk/5390324984c43f1214b8abf731ad495ba2df5341 -> trunk/5390324984c43f1214b8abf731ad495ba2df5341 2025-10-10T00:37:13.3672069Z * [new tag] trunk/53f5af8c924aba3c0fab1fabc6baf7d6affcb8a1 -> trunk/53f5af8c924aba3c0fab1fabc6baf7d6affcb8a1 2025-10-10T00:37:13.3672299Z * [new tag] trunk/54ae61c573e91fa2a2c6430435059e2d94ecba2e -> trunk/54ae61c573e91fa2a2c6430435059e2d94ecba2e 2025-10-10T00:37:13.3672542Z * [new tag] trunk/5656d45c8ff03cf20fd7d5098247c2250395af8a -> trunk/5656d45c8ff03cf20fd7d5098247c2250395af8a 2025-10-10T00:37:13.3672783Z * [new tag] trunk/56d66ac0d74f44d7b656757795142b5b9a1802a1 -> trunk/56d66ac0d74f44d7b656757795142b5b9a1802a1 2025-10-10T00:37:13.3673033Z * [new tag] trunk/5743d731c1de495ecf3bb03682a2dcbe207ca895 -> trunk/5743d731c1de495ecf3bb03682a2dcbe207ca895 2025-10-10T00:37:13.3673281Z * [new tag] trunk/5a1fbf45ad727353e367740ecd8825ca7ee857e9 -> trunk/5a1fbf45ad727353e367740ecd8825ca7ee857e9 2025-10-10T00:37:13.3673563Z * [new tag] trunk/5a66ff4915ecfd86f1a68e7862e5a2ad473e5a79 -> trunk/5a66ff4915ecfd86f1a68e7862e5a2ad473e5a79 2025-10-10T00:37:13.3673804Z * [new tag] trunk/5b0b4cda4aa03bee16ee67d9d36012a539df3c50 -> trunk/5b0b4cda4aa03bee16ee67d9d36012a539df3c50 2025-10-10T00:37:13.3674027Z * [new tag] trunk/5b8174bc286725f9326fba6dc0ef17c316486bbd -> trunk/5b8174bc286725f9326fba6dc0ef17c316486bbd 2025-10-10T00:37:13.3674259Z * [new tag] trunk/5ba11df4f871717818b88c4eab514d31286601d1 -> trunk/5ba11df4f871717818b88c4eab514d31286601d1 2025-10-10T00:37:13.3674483Z * [new tag] trunk/5c827a4133da69108338d0363bb7ad7f62803c40 -> trunk/5c827a4133da69108338d0363bb7ad7f62803c40 2025-10-10T00:37:13.3674703Z * [new tag] trunk/5d459dd6099ef94d33db9a6d36bcce9f742f1da1 -> trunk/5d459dd6099ef94d33db9a6d36bcce9f742f1da1 2025-10-10T00:37:13.3674927Z * [new tag] trunk/5d7360bb03355c89c0b956df0ab428f5a7b5c9f8 -> trunk/5d7360bb03355c89c0b956df0ab428f5a7b5c9f8 2025-10-10T00:37:13.3675150Z * [new tag] trunk/5dbae1eae26159058f6199fc68fe73fc0e5bef5f -> trunk/5dbae1eae26159058f6199fc68fe73fc0e5bef5f 2025-10-10T00:37:13.3675375Z * [new tag] trunk/5e47b4dd60ff9efb253286af5a2479d9d800ce6a -> trunk/5e47b4dd60ff9efb253286af5a2479d9d800ce6a 2025-10-10T00:37:13.3675595Z * [new tag] trunk/5ed4270440fd0b62d3aa14692f9e377a0061061e -> trunk/5ed4270440fd0b62d3aa14692f9e377a0061061e 2025-10-10T00:37:13.3675833Z * [new tag] trunk/5f18f240de43fc24481ead4d740dda64f174fa86 -> trunk/5f18f240de43fc24481ead4d740dda64f174fa86 2025-10-10T00:37:13.3676071Z * [new tag] trunk/5f775bdfb766d9a2717ffbb64f2a51e53cddc778 -> trunk/5f775bdfb766d9a2717ffbb64f2a51e53cddc778 2025-10-10T00:37:13.3676288Z * [new tag] trunk/600267ea56cafcf8f9a1150a4379184960a757b2 -> trunk/600267ea56cafcf8f9a1150a4379184960a757b2 2025-10-10T00:37:13.3676516Z * [new tag] trunk/600db525bdb5e76c12f30f271d969d43a7f8efef -> trunk/600db525bdb5e76c12f30f271d969d43a7f8efef 2025-10-10T00:37:13.3676730Z * [new tag] trunk/608792153f42254d2d2b5a87d524807a0c2724f1 -> trunk/608792153f42254d2d2b5a87d524807a0c2724f1 2025-10-10T00:37:13.3676949Z * [new tag] trunk/6389658ec6b1ea58cb1de032266d865eeb8d48e9 -> trunk/6389658ec6b1ea58cb1de032266d865eeb8d48e9 2025-10-10T00:37:13.3677200Z * [new tag] trunk/64108bdbed2f099d527060b4c9fdd5a11cad2afc -> trunk/64108bdbed2f099d527060b4c9fdd5a11cad2afc 2025-10-10T00:37:13.3677427Z * [new tag] trunk/65aa62d50d1c83aa1b46ed4d584f12f509bab1c4 -> trunk/65aa62d50d1c83aa1b46ed4d584f12f509bab1c4 2025-10-10T00:37:13.3677663Z * [new tag] trunk/65f10becdf21f3a0947a735904fcce876ce3c4b0 -> trunk/65f10becdf21f3a0947a735904fcce876ce3c4b0 2025-10-10T00:37:13.3677902Z * [new tag] trunk/660e369a68dd8be60ce4eb67c25191ea66efc303 -> trunk/660e369a68dd8be60ce4eb67c25191ea66efc303 2025-10-10T00:37:13.3678140Z * [new tag] trunk/68350660ee2db8c21c84527929b92de9f0bcc3e2 -> trunk/68350660ee2db8c21c84527929b92de9f0bcc3e2 2025-10-10T00:37:13.3678349Z * [new tag] trunk/6861a270624b44954826688f8dad668eb0154452 -> trunk/6861a270624b44954826688f8dad668eb0154452 2025-10-10T00:37:13.3678583Z * [new tag] trunk/6861fa43e5fee7fedc0213e352fa983edea8aa78 -> trunk/6861fa43e5fee7fedc0213e352fa983edea8aa78 2025-10-10T00:37:13.3678806Z * [new tag] trunk/688efd9741dbd18c176729aec3df7a73825f8463 -> trunk/688efd9741dbd18c176729aec3df7a73825f8463 2025-10-10T00:37:13.3679036Z * [new tag] trunk/6a09f9306cadd003b2e6abc3f6422a2d8607779b -> trunk/6a09f9306cadd003b2e6abc3f6422a2d8607779b 2025-10-10T00:37:13.3679259Z * [new tag] trunk/6a31f42da45c0f1cbdb021b3695f0e6388b8b532 -> trunk/6a31f42da45c0f1cbdb021b3695f0e6388b8b532 2025-10-10T00:37:13.3679522Z * [new tag] trunk/6a7f5c0d21a22959d014c8b06f3efe3408336aaf -> trunk/6a7f5c0d21a22959d014c8b06f3efe3408336aaf 2025-10-10T00:37:13.3679737Z * [new tag] trunk/6b768e1890a179122e91395c5532a382d69b96a0 -> trunk/6b768e1890a179122e91395c5532a382d69b96a0 2025-10-10T00:37:13.3679970Z * [new tag] trunk/6b7970192f5de47d29a4fe085f509389ac0bea7d -> trunk/6b7970192f5de47d29a4fe085f509389ac0bea7d 2025-10-10T00:37:13.3680215Z * [new tag] trunk/6bb021c12553755a4f64df0b60dc34b1efdb992b -> trunk/6bb021c12553755a4f64df0b60dc34b1efdb992b 2025-10-10T00:37:13.3680462Z * [new tag] trunk/6bb586eafd723d4972c729f37c14f27c88168adc -> trunk/6bb586eafd723d4972c729f37c14f27c88168adc 2025-10-10T00:37:13.3683124Z * [new tag] trunk/6c0125dbc0241aef962528651df4f67204a8b526 -> trunk/6c0125dbc0241aef962528651df4f67204a8b526 2025-10-10T00:37:13.3688469Z * [new tag] trunk/6c209bfc5c1e1e59e6a62f94151398d66164bb93 -> trunk/6c209bfc5c1e1e59e6a62f94151398d66164bb93 2025-10-10T00:37:13.3688808Z * [new tag] trunk/6c3c9414eb571b34ff0d932978e4733dbb08dc1d -> trunk/6c3c9414eb571b34ff0d932978e4733dbb08dc1d 2025-10-10T00:37:13.3689093Z * [new tag] trunk/6d27a8e5093ee2a21d44dceeeffcb272e6e0f655 -> trunk/6d27a8e5093ee2a21d44dceeeffcb272e6e0f655 2025-10-10T00:37:13.3689358Z * [new tag] trunk/702f6e703b1d3a942346848b65a9f2a37d12ae18 -> trunk/702f6e703b1d3a942346848b65a9f2a37d12ae18 2025-10-10T00:37:13.3689634Z * [new tag] trunk/7158aa22e8dc97fdc2657cf0d4cde34b277e7d9e -> trunk/7158aa22e8dc97fdc2657cf0d4cde34b277e7d9e 2025-10-10T00:37:13.3689886Z * [new tag] trunk/71aefd5595834dd97f38aa978ee32abbd13ac3d6 -> trunk/71aefd5595834dd97f38aa978ee32abbd13ac3d6 2025-10-10T00:37:13.3690137Z * [new tag] trunk/724463d5a2fba369cd14e89215b84d1b01435df7 -> trunk/724463d5a2fba369cd14e89215b84d1b01435df7 2025-10-10T00:37:13.3690423Z * [new tag] trunk/73adac05d13babb75410c3e033fdce57aa16881a -> trunk/73adac05d13babb75410c3e033fdce57aa16881a 2025-10-10T00:37:13.3690645Z * [new tag] trunk/7457d139c51124e5a31a6173f99f81f0deb52178 -> trunk/7457d139c51124e5a31a6173f99f81f0deb52178 2025-10-10T00:37:13.3690890Z * [new tag] trunk/746fe78ecd52f3e9cfddda41f0ac82dada7bdd0b -> trunk/746fe78ecd52f3e9cfddda41f0ac82dada7bdd0b 2025-10-10T00:37:13.3691122Z * [new tag] trunk/7617b113ad0045cdfe5cf1feb8efb634a41c6ce2 -> trunk/7617b113ad0045cdfe5cf1feb8efb634a41c6ce2 2025-10-10T00:37:13.3691513Z * [new tag] trunk/7a1ead755f2e2abe8be49a7a0fb88b6b13973147 -> trunk/7a1ead755f2e2abe8be49a7a0fb88b6b13973147 2025-10-10T00:37:13.3691744Z * [new tag] trunk/7b15534434aeaf59a4c9189f52b4ebd4a5d58803 -> trunk/7b15534434aeaf59a4c9189f52b4ebd4a5d58803 2025-10-10T00:37:13.3691972Z * [new tag] trunk/7b691546d2949790ffc8f6bd3c674faa6a46ff7c -> trunk/7b691546d2949790ffc8f6bd3c674faa6a46ff7c 2025-10-10T00:37:13.3692213Z * [new tag] trunk/7cfecd76b2141d81c90d722dc5e3262bdf7ea900 -> trunk/7cfecd76b2141d81c90d722dc5e3262bdf7ea900 2025-10-10T00:37:13.3692447Z * [new tag] trunk/7d570129e0cea8dd3de0175baff96723656ab8ab -> trunk/7d570129e0cea8dd3de0175baff96723656ab8ab 2025-10-10T00:37:13.3692694Z * [new tag] trunk/7e7ac2039d5d5f35373c4de6cdf0ccdee3734c7a -> trunk/7e7ac2039d5d5f35373c4de6cdf0ccdee3734c7a 2025-10-10T00:37:13.3692933Z * [new tag] trunk/7eb1eb4313cfa3db1beadc6d9d04ea6b76acc39c -> trunk/7eb1eb4313cfa3db1beadc6d9d04ea6b76acc39c 2025-10-10T00:37:13.3693177Z * [new tag] trunk/801e282f39e9ef4424dfd3ecfd2b550a44595229 -> trunk/801e282f39e9ef4424dfd3ecfd2b550a44595229 2025-10-10T00:37:13.3693399Z * [new tag] trunk/81994b08a078b30e076d408713f78c9bf4e329e7 -> trunk/81994b08a078b30e076d408713f78c9bf4e329e7 2025-10-10T00:37:13.3693640Z * [new tag] trunk/81dbeb06f4b3eb6c56625ec25d377eb7c7c6c573 -> trunk/81dbeb06f4b3eb6c56625ec25d377eb7c7c6c573 2025-10-10T00:37:13.3693960Z * [new tag] trunk/83458197d14921f797565135f0f45031c362338d -> trunk/83458197d14921f797565135f0f45031c362338d 2025-10-10T00:37:13.3694210Z * [new tag] trunk/83d71dfb2fd993a6242372b8123549acaa85ffdb -> trunk/83d71dfb2fd993a6242372b8123549acaa85ffdb 2025-10-10T00:37:13.3694457Z * [new tag] trunk/86474ce996d168b404592cbbdfcc30d6607c8bd4 -> trunk/86474ce996d168b404592cbbdfcc30d6607c8bd4 2025-10-10T00:37:13.3694711Z * [new tag] trunk/86c789849eac1f96d03cf273e7995dbc7d319c26 -> trunk/86c789849eac1f96d03cf273e7995dbc7d319c26 2025-10-10T00:37:13.3694971Z * [new tag] trunk/874efa2d72d83b00894097130f18062ce331a265 -> trunk/874efa2d72d83b00894097130f18062ce331a265 2025-10-10T00:37:13.3695222Z * [new tag] trunk/87c9fbda22c229d4e5512011e050efd6ffea1241 -> trunk/87c9fbda22c229d4e5512011e050efd6ffea1241 2025-10-10T00:37:13.3695508Z * [new tag] trunk/87eccf10e8484c9e59ef81ae7bdee68d3db4f605 -> trunk/87eccf10e8484c9e59ef81ae7bdee68d3db4f605 2025-10-10T00:37:13.3695760Z * [new tag] trunk/8c0bc879b97bc580aaa0777b2d266bdd068cb528 -> trunk/8c0bc879b97bc580aaa0777b2d266bdd068cb528 2025-10-10T00:37:13.3696013Z * [new tag] trunk/8c54101933bb7c6ed3f9c1a65629b7f30376f7e2 -> trunk/8c54101933bb7c6ed3f9c1a65629b7f30376f7e2 2025-10-10T00:37:13.3696299Z * [new tag] trunk/8ca986ee60febce075f9e3ff83726048cebbbf68 -> trunk/8ca986ee60febce075f9e3ff83726048cebbbf68 2025-10-10T00:37:13.3696525Z * [new tag] trunk/8d53d788fefc0370931063d91f0c342556c3cf4c -> trunk/8d53d788fefc0370931063d91f0c342556c3cf4c 2025-10-10T00:37:13.3697779Z * [new tag] trunk/8e1f409b8ccf64b2cf3933ece13587ad57e9d8a9 -> trunk/8e1f409b8ccf64b2cf3933ece13587ad57e9d8a9 2025-10-10T00:37:13.3698048Z * [new tag] trunk/8ec8c14aced9f3e7ff4ab663822bed792d6c34f4 -> trunk/8ec8c14aced9f3e7ff4ab663822bed792d6c34f4 2025-10-10T00:37:13.3698313Z * [new tag] trunk/8f54e27e5decf41222f5d744069eb6572dbf275f -> trunk/8f54e27e5decf41222f5d744069eb6572dbf275f 2025-10-10T00:37:13.3698558Z * [new tag] trunk/8f705d019a64b1ca882e043b3eb98559273a9e59 -> trunk/8f705d019a64b1ca882e043b3eb98559273a9e59 2025-10-10T00:37:13.3698816Z * [new tag] trunk/8f83b3e71cb2af6244971af59bfbb6e2abb55f24 -> trunk/8f83b3e71cb2af6244971af59bfbb6e2abb55f24 2025-10-10T00:37:13.3699119Z * [new tag] trunk/90b4e130d6871bee4e1f15bb8294c1bbbf8f4ba5 -> trunk/90b4e130d6871bee4e1f15bb8294c1bbbf8f4ba5 2025-10-10T00:37:13.3702149Z * [new tag] trunk/90c0825e2deb0a46faf5cc2deb7184f6f8ea7a6d -> trunk/90c0825e2deb0a46faf5cc2deb7184f6f8ea7a6d 2025-10-10T00:37:13.3702383Z * [new tag] trunk/91040f49348646d79c6cd3434c34860d25c2e47a -> trunk/91040f49348646d79c6cd3434c34860d25c2e47a 2025-10-10T00:37:13.3702632Z * [new tag] trunk/91b94842645c1a781ab169b0df718545901ebb01 -> trunk/91b94842645c1a781ab169b0df718545901ebb01 2025-10-10T00:37:13.3702880Z * [new tag] trunk/91c211fb8c8ec3065be2a18dfc399ce849ea83bf -> trunk/91c211fb8c8ec3065be2a18dfc399ce849ea83bf 2025-10-10T00:37:13.3703137Z * [new tag] trunk/91c4db76cbb82dfa46d937b8dce4c942eaf5e226 -> trunk/91c4db76cbb82dfa46d937b8dce4c942eaf5e226 2025-10-10T00:37:13.3703373Z * [new tag] trunk/93e833de0f987f66d8c93b76ffe6aad35b714231 -> trunk/93e833de0f987f66d8c93b76ffe6aad35b714231 2025-10-10T00:37:13.3703611Z * [new tag] trunk/94b1ec8c7c5cc63541325abc923973f2fc2ad094 -> trunk/94b1ec8c7c5cc63541325abc923973f2fc2ad094 2025-10-10T00:37:13.3703855Z * [new tag] trunk/955f21dc2c628e09e0d112b3db1ee928cd1da344 -> trunk/955f21dc2c628e09e0d112b3db1ee928cd1da344 2025-10-10T00:37:13.3704094Z * [new tag] trunk/9580539e2f73d68e89544c713ff460bea3038701 -> trunk/9580539e2f73d68e89544c713ff460bea3038701 2025-10-10T00:37:13.3704389Z * [new tag] trunk/95a053284cd28e8d52bd55049bd45aea47adba0c -> trunk/95a053284cd28e8d52bd55049bd45aea47adba0c 2025-10-10T00:37:13.3704631Z * [new tag] trunk/960c4b9937251da01ea588efff0fc06a34eac35b -> trunk/960c4b9937251da01ea588efff0fc06a34eac35b 2025-10-10T00:37:13.3704887Z * [new tag] trunk/96181d6f7619acf938dc743123326c6b5dd25284 -> trunk/96181d6f7619acf938dc743123326c6b5dd25284 2025-10-10T00:37:13.3706142Z * [new tag] trunk/9697a7ce9ea095e933658cfee13f9bbef272551a -> trunk/9697a7ce9ea095e933658cfee13f9bbef272551a 2025-10-10T00:37:13.3706575Z * [new tag] trunk/96d91da792d4b50930318ecdfb8b5b8190c467cd -> trunk/96d91da792d4b50930318ecdfb8b5b8190c467cd 2025-10-10T00:37:13.3706825Z * [new tag] trunk/97463d4cf3c125557ef23502772b12a67dac4dc7 -> trunk/97463d4cf3c125557ef23502772b12a67dac4dc7 2025-10-10T00:37:13.3707075Z * [new tag] trunk/97ca21106d0179f425fc752ec867fe11669c2834 -> trunk/97ca21106d0179f425fc752ec867fe11669c2834 2025-10-10T00:37:13.3707335Z * [new tag] trunk/98a081a24c22072362dc536afd39a469e28939d4 -> trunk/98a081a24c22072362dc536afd39a469e28939d4 2025-10-10T00:37:13.3707567Z * [new tag] trunk/9944cac6e6a95159744a775a8bef40d89eef0f03 -> trunk/9944cac6e6a95159744a775a8bef40d89eef0f03 2025-10-10T00:37:13.3710091Z * [new tag] trunk/9aa92f246fa5fe5cfda17970d41d167b19a0612a -> trunk/9aa92f246fa5fe5cfda17970d41d167b19a0612a 2025-10-10T00:37:13.3710344Z * [new tag] trunk/9d1ab4f4bb508a72c7f549f0b5219c4601944ba1 -> trunk/9d1ab4f4bb508a72c7f549f0b5219c4601944ba1 2025-10-10T00:37:13.3710593Z * [new tag] trunk/9eb89a4ad5965b97c54e498d71fc765c0059acef -> trunk/9eb89a4ad5965b97c54e498d71fc765c0059acef 2025-10-10T00:37:13.3710837Z * [new tag] trunk/9ec10dc26a81dc618ff435edd4ca4819245ecb0f -> trunk/9ec10dc26a81dc618ff435edd4ca4819245ecb0f 2025-10-10T00:37:13.3711097Z * [new tag] trunk/9ecd092bd98f43d1cd4acc88eed6cbc39e946dbe -> trunk/9ecd092bd98f43d1cd4acc88eed6cbc39e946dbe 2025-10-10T00:37:13.3711337Z * [new tag] trunk/9f5e1beaf3c9248a335d2448103240a463187eb5 -> trunk/9f5e1beaf3c9248a335d2448103240a463187eb5 2025-10-10T00:37:13.3713785Z * [new tag] trunk/9fc2c6446d394dd313ed71e9d1ffc4f7f3916423 -> trunk/9fc2c6446d394dd313ed71e9d1ffc4f7f3916423 2025-10-10T00:37:13.3714021Z * [new tag] trunk/9fff8155c362da777e7ce31b85fb2dc7cfced2d5 -> trunk/9fff8155c362da777e7ce31b85fb2dc7cfced2d5 2025-10-10T00:37:13.3714293Z * [new tag] trunk/a029675f6f0b9cf48eb7943d4be8169c67960a8e -> trunk/a029675f6f0b9cf48eb7943d4be8169c67960a8e 2025-10-10T00:37:13.3714525Z * [new tag] trunk/a11a66ef320938cd0fd72b44b2b572b06937e100 -> trunk/a11a66ef320938cd0fd72b44b2b572b06937e100 2025-10-10T00:37:13.3714755Z * [new tag] trunk/a2f29bcd6388acdc3202d8a90974c50ffb605104 -> trunk/a2f29bcd6388acdc3202d8a90974c50ffb605104 2025-10-10T00:37:13.3714989Z * [new tag] trunk/a34797e031727f6a01a2f13a66db2f7e1fcc05b6 -> trunk/a34797e031727f6a01a2f13a66db2f7e1fcc05b6 2025-10-10T00:37:13.3721346Z * [new tag] trunk/a4110fedcf72eaede76324bb5c21a76589d75849 -> trunk/a4110fedcf72eaede76324bb5c21a76589d75849 2025-10-10T00:37:13.3726586Z * [new tag] trunk/a43c4c3972a611db169dde2aed803b91fe78c081 -> trunk/a43c4c3972a611db169dde2aed803b91fe78c081 2025-10-10T00:37:13.3731837Z * [new tag] trunk/a57a14868dcfd9dabf9bd19b6b11f31967c80c87 -> trunk/a57a14868dcfd9dabf9bd19b6b11f31967c80c87 2025-10-10T00:37:13.3737121Z * [new tag] trunk/a6fa4f9c283971c0fb6f60a89674a1f35370ac79 -> trunk/a6fa4f9c283971c0fb6f60a89674a1f35370ac79 2025-10-10T00:37:13.3742351Z * [new tag] trunk/a753ffa9aff47e005c31d6bcbf5b6a61cc54afed -> trunk/a753ffa9aff47e005c31d6bcbf5b6a61cc54afed 2025-10-10T00:37:13.3742684Z * [new tag] trunk/a7fa1a91e386c7708e4c8747680911b0c3174a66 -> trunk/a7fa1a91e386c7708e4c8747680911b0c3174a66 2025-10-10T00:37:13.3743189Z * [new tag] trunk/a9a9a3438a374f96a308b707a1718036aaec790d -> trunk/a9a9a3438a374f96a308b707a1718036aaec790d 2025-10-10T00:37:13.3743459Z * [new tag] trunk/ab01a0d7d352e7fd07989b8d6bf035bf82aea74e -> trunk/ab01a0d7d352e7fd07989b8d6bf035bf82aea74e 2025-10-10T00:37:13.3743716Z * [new tag] trunk/ab94a0d544503b5c27e889b45e45ef8cf75c8183 -> trunk/ab94a0d544503b5c27e889b45e45ef8cf75c8183 2025-10-10T00:37:13.3743985Z * [new tag] trunk/abadea70f3eb5f2f764fd6448d42dd2c29fa28b3 -> trunk/abadea70f3eb5f2f764fd6448d42dd2c29fa28b3 2025-10-10T00:37:13.3744230Z * [new tag] trunk/ac08556f674259ff5b117964e300124e8a92d45b -> trunk/ac08556f674259ff5b117964e300124e8a92d45b 2025-10-10T00:37:13.3744481Z * [new tag] trunk/ac7b4e7fe4d233dcd7f6343d42b4fa3d64bce548 -> trunk/ac7b4e7fe4d233dcd7f6343d42b4fa3d64bce548 2025-10-10T00:37:13.3744743Z * [new tag] trunk/ac901bf79a2d78539ffec272bf32f4ae47035b23 -> trunk/ac901bf79a2d78539ffec272bf32f4ae47035b23 2025-10-10T00:37:13.3745015Z * [new tag] trunk/ad7b2bebc651c297d869f265deedef726bf17048 -> trunk/ad7b2bebc651c297d869f265deedef726bf17048 2025-10-10T00:37:13.3745279Z * [new tag] trunk/ae25ec569c614c2a2274837079578b71f3201a3b -> trunk/ae25ec569c614c2a2274837079578b71f3201a3b 2025-10-10T00:37:13.3745537Z * [new tag] trunk/aea57b3aa38a3d4a058e0a7eba08d0c6c28ed9c5 -> trunk/aea57b3aa38a3d4a058e0a7eba08d0c6c28ed9c5 2025-10-10T00:37:13.3745796Z * [new tag] trunk/aed5ed1076d3e73e0b6357dafac1002aa6a221e9 -> trunk/aed5ed1076d3e73e0b6357dafac1002aa6a221e9 2025-10-10T00:37:13.3746057Z * [new tag] trunk/aed66248a01d309eb2ac1149b5f51310545b0783 -> trunk/aed66248a01d309eb2ac1149b5f51310545b0783 2025-10-10T00:37:13.3746524Z * [new tag] trunk/af32d16a71681ca05c6d410fb1b9cee091d4577d -> trunk/af32d16a71681ca05c6d410fb1b9cee091d4577d 2025-10-10T00:37:13.3746797Z * [new tag] trunk/af40828bbb785f968eda18dbdc8750ba67f57366 -> trunk/af40828bbb785f968eda18dbdc8750ba67f57366 2025-10-10T00:37:13.3747052Z * [new tag] trunk/af4c29fea8f50ac3bb9e4a0e305da4a2c6b53d29 -> trunk/af4c29fea8f50ac3bb9e4a0e305da4a2c6b53d29 2025-10-10T00:37:13.3747305Z * [new tag] trunk/afee8062d511ad63e0af65ffac0e712d86aae8f1 -> trunk/afee8062d511ad63e0af65ffac0e712d86aae8f1 2025-10-10T00:37:13.3747654Z * [new tag] trunk/afeec56a5aa83dd0258565400551a99777c0023b -> trunk/afeec56a5aa83dd0258565400551a99777c0023b 2025-10-10T00:37:13.3747902Z * [new tag] trunk/b0985144b59db8fb20964829b5e0a9d2f9a3f0d6 -> trunk/b0985144b59db8fb20964829b5e0a9d2f9a3f0d6 2025-10-10T00:37:13.3748157Z * [new tag] trunk/b116c5133024be39a2db67cd0112b490b970b710 -> trunk/b116c5133024be39a2db67cd0112b490b970b710 2025-10-10T00:37:13.3748413Z * [new tag] trunk/b13cd141b3585c2ae89ad7747acd11203a2fb837 -> trunk/b13cd141b3585c2ae89ad7747acd11203a2fb837 2025-10-10T00:37:13.3748686Z * [new tag] trunk/b1ac252f55f4a4d0e5488fb2ac9154154decec87 -> trunk/b1ac252f55f4a4d0e5488fb2ac9154154decec87 2025-10-10T00:37:13.3748931Z * [new tag] trunk/b28b24a9fc7d391c5793a94489a3f2d5381f6ad7 -> trunk/b28b24a9fc7d391c5793a94489a3f2d5381f6ad7 2025-10-10T00:37:13.3749233Z * [new tag] trunk/b2b3947565fd0c27ebd4941152c964eab30370e2 -> trunk/b2b3947565fd0c27ebd4941152c964eab30370e2 2025-10-10T00:37:13.3749481Z * [new tag] trunk/b558c986e8ec693b531ad2817026393c55d72eb6 -> trunk/b558c986e8ec693b531ad2817026393c55d72eb6 2025-10-10T00:37:13.3749744Z * [new tag] trunk/b5e93ffdcf779c703af5c8119636b01f250eafcd -> trunk/b5e93ffdcf779c703af5c8119636b01f250eafcd 2025-10-10T00:37:13.3750008Z * [new tag] trunk/b63bbe16615cc7680836dbb151bd848bce4893d6 -> trunk/b63bbe16615cc7680836dbb151bd848bce4893d6 2025-10-10T00:37:13.3750313Z * [new tag] trunk/b6b7a44dec63495d57946cbfe8f2accb8f876db2 -> trunk/b6b7a44dec63495d57946cbfe8f2accb8f876db2 2025-10-10T00:37:13.3750559Z * [new tag] trunk/b9e73e639e36f3aa628752161711e68878231b30 -> trunk/b9e73e639e36f3aa628752161711e68878231b30 2025-10-10T00:37:13.3750815Z * [new tag] trunk/ba480d6bf78ea446d1268d9b5b3a0dbb490c9c88 -> trunk/ba480d6bf78ea446d1268d9b5b3a0dbb490c9c88 2025-10-10T00:37:13.3751065Z * [new tag] trunk/bac0f289a35f05052740076fc5671271a3d487c2 -> trunk/bac0f289a35f05052740076fc5671271a3d487c2 2025-10-10T00:37:13.3751322Z * [new tag] trunk/bc1690c7e859dee8c47a7f0bbd3c43cc27c6fd2a -> trunk/bc1690c7e859dee8c47a7f0bbd3c43cc27c6fd2a 2025-10-10T00:37:13.3751567Z * [new tag] trunk/bc33b10202fb7c3761bcabc166e02d96807d8739 -> trunk/bc33b10202fb7c3761bcabc166e02d96807d8739 2025-10-10T00:37:13.3751861Z * [new tag] trunk/bcafea5c92ca2ee1b0dc8f6d8b62ecabb6f40228 -> trunk/bcafea5c92ca2ee1b0dc8f6d8b62ecabb6f40228 2025-10-10T00:37:13.3752130Z * [new tag] trunk/bcd96cc6ff798281e66aabef6ce72542fdc97c7a -> trunk/bcd96cc6ff798281e66aabef6ce72542fdc97c7a 2025-10-10T00:37:13.3752388Z * [new tag] trunk/bd3b98a8a5d68ddc84b20a4609b9ea90998bf95b -> trunk/bd3b98a8a5d68ddc84b20a4609b9ea90998bf95b 2025-10-10T00:37:13.3752636Z * [new tag] trunk/bdc0a421d7bcc49db12f7593d2c213a6141da614 -> trunk/bdc0a421d7bcc49db12f7593d2c213a6141da614 2025-10-10T00:37:13.3752898Z * [new tag] trunk/bde18c445dcb1d83e8ea0afae52f9b9bf8171f45 -> trunk/bde18c445dcb1d83e8ea0afae52f9b9bf8171f45 2025-10-10T00:37:13.3753153Z * [new tag] trunk/bf717ce346203fc27e792f4bdcc31e979cd74fa9 -> trunk/bf717ce346203fc27e792f4bdcc31e979cd74fa9 2025-10-10T00:37:13.3753405Z * [new tag] trunk/c0510dc447a1f105cb8758d2721380f7a7c380d1 -> trunk/c0510dc447a1f105cb8758d2721380f7a7c380d1 2025-10-10T00:37:13.3753666Z * [new tag] trunk/c1f40d33c89b361a1edad17aa25cfff1ab4014fd -> trunk/c1f40d33c89b361a1edad17aa25cfff1ab4014fd 2025-10-10T00:37:13.3753911Z * [new tag] trunk/c32118dc3e50505fd285e6e448a90883fce11535 -> trunk/c32118dc3e50505fd285e6e448a90883fce11535 2025-10-10T00:37:13.3754148Z * [new tag] trunk/c45d56dd00546daa7d9044674233dba1ac7b6194 -> trunk/c45d56dd00546daa7d9044674233dba1ac7b6194 2025-10-10T00:37:13.3754395Z * [new tag] trunk/c6329524d8670d5f9295cddcf7ebc3040ed9179e -> trunk/c6329524d8670d5f9295cddcf7ebc3040ed9179e 2025-10-10T00:37:13.3754693Z * [new tag] trunk/c6a6c80a730ff4edaec0d2fc4a5ff9344edaed41 -> trunk/c6a6c80a730ff4edaec0d2fc4a5ff9344edaed41 2025-10-10T00:37:13.3754960Z * [new tag] trunk/c7e30ae4dd9a58ed4f4bcbdc6afc2249cac94f28 -> trunk/c7e30ae4dd9a58ed4f4bcbdc6afc2249cac94f28 2025-10-10T00:37:13.3755211Z * [new tag] trunk/c813617c53e6be91e77f47e9a3f713146d54f340 -> trunk/c813617c53e6be91e77f47e9a3f713146d54f340 2025-10-10T00:37:13.3755453Z * [new tag] trunk/c855f8632e331b51d60d5f1bcc59d3181cb4bc82 -> trunk/c855f8632e331b51d60d5f1bcc59d3181cb4bc82 2025-10-10T00:37:13.3755720Z * [new tag] trunk/c965d6dbb23a8b2338ffebf3f01c6f92ce5847d2 -> trunk/c965d6dbb23a8b2338ffebf3f01c6f92ce5847d2 2025-10-10T00:37:13.3756020Z * [new tag] trunk/cac5e13e1384900c5acc4938c33d6037a61850d5 -> trunk/cac5e13e1384900c5acc4938c33d6037a61850d5 2025-10-10T00:37:13.3756263Z * [new tag] trunk/cc71ab86a6985e85645424b727c766e031047ff6 -> trunk/cc71ab86a6985e85645424b727c766e031047ff6 2025-10-10T00:37:13.3756565Z * [new tag] trunk/cd62a73dcb13102069aa827a6657f62d88cce095 -> trunk/cd62a73dcb13102069aa827a6657f62d88cce095 2025-10-10T00:37:13.3756781Z * [new tag] trunk/cf0a00d4f38775e5a82a166e367f40383c606963 -> trunk/cf0a00d4f38775e5a82a166e367f40383c606963 2025-10-10T00:37:13.3757015Z * [new tag] trunk/cfc5cc17dc4fa6be41b4b31eb6e63d3863479452 -> trunk/cfc5cc17dc4fa6be41b4b31eb6e63d3863479452 2025-10-10T00:37:13.3757274Z * [new tag] trunk/cfd46d13e6e1308add3a9f287b4855ccc3f2e66c -> trunk/cfd46d13e6e1308add3a9f287b4855ccc3f2e66c 2025-10-10T00:37:13.3757494Z * [new tag] trunk/d1a62c80363cf769552453eed187e935f905737d -> trunk/d1a62c80363cf769552453eed187e935f905737d 2025-10-10T00:37:13.3757706Z * [new tag] trunk/d1cbb74fb16406488a174832e1b58b7c242f418d -> trunk/d1cbb74fb16406488a174832e1b58b7c242f418d 2025-10-10T00:37:13.3757928Z * [new tag] trunk/d386325ca9a142419f45b987391f4bb175dd7d0b -> trunk/d386325ca9a142419f45b987391f4bb175dd7d0b 2025-10-10T00:37:13.3758149Z * [new tag] trunk/d40a9bfb8da0dc1ac1e6e56b33a25979112874de -> trunk/d40a9bfb8da0dc1ac1e6e56b33a25979112874de 2025-10-10T00:37:13.3758359Z * [new tag] trunk/d4443840036a00a30afcf066cb23f4525e590809 -> trunk/d4443840036a00a30afcf066cb23f4525e590809 2025-10-10T00:37:13.3758589Z * [new tag] trunk/d4752bc7f6818a3df5356a9de61afe1d3e27ade9 -> trunk/d4752bc7f6818a3df5356a9de61afe1d3e27ade9 2025-10-10T00:37:13.3758803Z * [new tag] trunk/da49a57d3462332b26cb7ee58910b5bc67e5772c -> trunk/da49a57d3462332b26cb7ee58910b5bc67e5772c 2025-10-10T00:37:13.3759029Z * [new tag] trunk/da903b6a8be422529d47649e89c0d50bb95c37ca -> trunk/da903b6a8be422529d47649e89c0d50bb95c37ca 2025-10-10T00:37:13.3759434Z * [new tag] trunk/dca73982c53e9f99f96246b5d9ed9bab83c7423f -> trunk/dca73982c53e9f99f96246b5d9ed9bab83c7423f 2025-10-10T00:37:13.3759744Z * [new tag] trunk/ddf8de28c25944a58e739ba9996b06753e4199cc -> trunk/ddf8de28c25944a58e739ba9996b06753e4199cc 2025-10-10T00:37:13.3760053Z * [new tag] trunk/df640df68a5275684eaae3080a9c97a0c61469c8 -> trunk/df640df68a5275684eaae3080a9c97a0c61469c8 2025-10-10T00:37:13.3760366Z * [new tag] trunk/e09fb44ef177005c4a11c28be24781429d416a3e -> trunk/e09fb44ef177005c4a11c28be24781429d416a3e 2025-10-10T00:37:13.3760671Z * [new tag] trunk/e0cb1848d0fd9fb4467ad8b844c565aea5071838 -> trunk/e0cb1848d0fd9fb4467ad8b844c565aea5071838 2025-10-10T00:37:13.3760991Z * [new tag] trunk/e3ae80fc036da356e3748d134689741583552f09 -> trunk/e3ae80fc036da356e3748d134689741583552f09 2025-10-10T00:37:13.3762010Z * [new tag] trunk/e40fe634b1a7aa33e278b1404ee02dea12277080 -> trunk/e40fe634b1a7aa33e278b1404ee02dea12277080 2025-10-10T00:37:13.3762593Z * [new tag] trunk/e438db254602cf39ba536aed0590b4144c019ee8 -> trunk/e438db254602cf39ba536aed0590b4144c019ee8 2025-10-10T00:37:13.3763264Z * [new tag] trunk/e532f62e0d96e56cb28fa6a0ba6d981896a65d52 -> trunk/e532f62e0d96e56cb28fa6a0ba6d981896a65d52 2025-10-10T00:37:13.3763510Z * [new tag] trunk/e6d4b26776842307475b368db60e27ac1bcede86 -> trunk/e6d4b26776842307475b368db60e27ac1bcede86 2025-10-10T00:37:13.3766454Z * [new tag] trunk/e7ed1a00eb5510d1c7dccd17b5c0ebb54231284f -> trunk/e7ed1a00eb5510d1c7dccd17b5c0ebb54231284f 2025-10-10T00:37:13.3766882Z * [new tag] trunk/e7fd2969303ab931f5e5875eca676018e1acd089 -> trunk/e7fd2969303ab931f5e5875eca676018e1acd089 2025-10-10T00:37:13.3767253Z * [new tag] trunk/e89d12bf5d6b69c153cd000ef278fca59f03226d -> trunk/e89d12bf5d6b69c153cd000ef278fca59f03226d 2025-10-10T00:37:13.3767631Z * [new tag] trunk/e98c4e835b1db22092fc93b49d2cddd7b3537d1f -> trunk/e98c4e835b1db22092fc93b49d2cddd7b3537d1f 2025-10-10T00:37:13.3768356Z * [new tag] trunk/ea42517e454d2e47391646bbb897f5fc51147b9d -> trunk/ea42517e454d2e47391646bbb897f5fc51147b9d 2025-10-10T00:37:13.3768647Z * [new tag] trunk/eaa02655eabd24609744c2251ac40d39d86ebb39 -> trunk/eaa02655eabd24609744c2251ac40d39d86ebb39 2025-10-10T00:37:13.3768918Z * [new tag] trunk/eccf561326147894d57482a5aba7a2290005b257 -> trunk/eccf561326147894d57482a5aba7a2290005b257 2025-10-10T00:37:13.3769395Z * [new tag] trunk/ece5e0f01b68509342f85fa388ca61936dc18b20 -> trunk/ece5e0f01b68509342f85fa388ca61936dc18b20 2025-10-10T00:37:13.3769843Z * [new tag] trunk/ed2d514ad860229f6d364688f9db27dad034cd83 -> trunk/ed2d514ad860229f6d364688f9db27dad034cd83 2025-10-10T00:37:13.3770149Z * [new tag] trunk/ed6156e3ea334b9b8d395e5a9f76fa3ba7408c06 -> trunk/ed6156e3ea334b9b8d395e5a9f76fa3ba7408c06 2025-10-10T00:37:13.3770505Z * [new tag] trunk/ee5389d520844db36374e86c986b9ff8f47ac4bb -> trunk/ee5389d520844db36374e86c986b9ff8f47ac4bb 2025-10-10T00:37:13.3770762Z * [new tag] trunk/ee6a1ecb0a1035f068484c8fcfba44b2efc9e837 -> trunk/ee6a1ecb0a1035f068484c8fcfba44b2efc9e837 2025-10-10T00:37:13.3771609Z * [new tag] trunk/ef50c6e3e3d83bfd67e50930eea9a3a9db084061 -> trunk/ef50c6e3e3d83bfd67e50930eea9a3a9db084061 2025-10-10T00:37:13.3772063Z * [new tag] trunk/ef7e2ca77e3f554ced81eb614f15fb84249d4a7e -> trunk/ef7e2ca77e3f554ced81eb614f15fb84249d4a7e 2025-10-10T00:37:13.3775188Z * [new tag] trunk/f006aee601cb72077f4b1dbc3f7f0f685e57a1a9 -> trunk/f006aee601cb72077f4b1dbc3f7f0f685e57a1a9 2025-10-10T00:37:13.3775491Z * [new tag] trunk/f05e23e1bc1439e19145e43e8ffca0051cda2f33 -> trunk/f05e23e1bc1439e19145e43e8ffca0051cda2f33 2025-10-10T00:37:13.3775761Z * [new tag] trunk/f0c9f3bddbf7ad77d5d3a8803c23bb47bfb71d79 -> trunk/f0c9f3bddbf7ad77d5d3a8803c23bb47bfb71d79 2025-10-10T00:37:13.3776054Z * [new tag] trunk/f11ac803d73b90d7e1f7bde962b9afe6b5967eb7 -> trunk/f11ac803d73b90d7e1f7bde962b9afe6b5967eb7 2025-10-10T00:37:13.3783343Z * [new tag] trunk/f1229b6db946c290cc5bdea05dde69fc01e0bed0 -> trunk/f1229b6db946c290cc5bdea05dde69fc01e0bed0 2025-10-10T00:37:13.3783636Z * [new tag] trunk/f231be25c679adb47ac3e483dc68948e5ad137a4 -> trunk/f231be25c679adb47ac3e483dc68948e5ad137a4 2025-10-10T00:37:13.3783941Z * [new tag] trunk/f33201729416ed17467228e80b04d01d4d02b5f3 -> trunk/f33201729416ed17467228e80b04d01d4d02b5f3 2025-10-10T00:37:13.3784221Z * [new tag] trunk/f37a6523efe1b9bf7f6b5b5d0f36dc461a3fda2a -> trunk/f37a6523efe1b9bf7f6b5b5d0f36dc461a3fda2a 2025-10-10T00:37:13.3784480Z * [new tag] trunk/f39789cdabb6465f21666bd001829e1f7284d754 -> trunk/f39789cdabb6465f21666bd001829e1f7284d754 2025-10-10T00:37:13.3784881Z * [new tag] trunk/f3afbcf3407783e54ec2795b06ae744f645320ba -> trunk/f3afbcf3407783e54ec2795b06ae744f645320ba 2025-10-10T00:37:13.3785144Z * [new tag] trunk/f3e43ff2d73f375487b1b71483bbecb6cdad8920 -> trunk/f3e43ff2d73f375487b1b71483bbecb6cdad8920 2025-10-10T00:37:13.3785410Z * [new tag] trunk/f414aa8e0d17e8eff38a93cebd52436e53f50eba -> trunk/f414aa8e0d17e8eff38a93cebd52436e53f50eba 2025-10-10T00:37:13.3785660Z * [new tag] trunk/f465ea6752c91498de63eb57439a74f4836e568a -> trunk/f465ea6752c91498de63eb57439a74f4836e568a 2025-10-10T00:37:13.3785929Z * [new tag] trunk/f46bb04dcc37a9b394e414569aef8aef69f9bf53 -> trunk/f46bb04dcc37a9b394e414569aef8aef69f9bf53 2025-10-10T00:37:13.3786189Z * [new tag] trunk/f46ddb1e65b595c80f285dc42aa8549970736aae -> trunk/f46ddb1e65b595c80f285dc42aa8549970736aae 2025-10-10T00:37:13.3786559Z * [new tag] trunk/f4cf75688f0fd93466589addfb7d0ec33e46e3bf -> trunk/f4cf75688f0fd93466589addfb7d0ec33e46e3bf 2025-10-10T00:37:13.3786820Z * [new tag] trunk/f505caa71bd2e4d1e708e20a3665b834134e08fc -> trunk/f505caa71bd2e4d1e708e20a3665b834134e08fc 2025-10-10T00:37:13.3787107Z * [new tag] trunk/f5fd18f7e24378bd9eb91404f697f1c81a8187d5 -> trunk/f5fd18f7e24378bd9eb91404f697f1c81a8187d5 2025-10-10T00:37:13.3787363Z * [new tag] trunk/f6de195616432f42a545b98ea41cc816019d1c60 -> trunk/f6de195616432f42a545b98ea41cc816019d1c60 2025-10-10T00:37:13.3787667Z * [new tag] trunk/f6f76767563d4293a0f78551edf4675a5794c570 -> trunk/f6f76767563d4293a0f78551edf4675a5794c570 2025-10-10T00:37:13.3787918Z * [new tag] trunk/f7082e92b3635e89906fae514506152a2ec844a0 -> trunk/f7082e92b3635e89906fae514506152a2ec844a0 2025-10-10T00:37:13.3788194Z * [new tag] trunk/f713abab16cb98c15f486e9822dd261279cce252 -> trunk/f713abab16cb98c15f486e9822dd261279cce252 2025-10-10T00:37:13.3788441Z * [new tag] trunk/f76fdcaaf8b6d5f97c7f63705400ebed8984f869 -> trunk/f76fdcaaf8b6d5f97c7f63705400ebed8984f869 2025-10-10T00:37:13.3793058Z * [new tag] trunk/f79e212733ca89ce3cc99a3072e50351686e5568 -> trunk/f79e212733ca89ce3cc99a3072e50351686e5568 2025-10-10T00:37:13.3797478Z * [new tag] trunk/f7ad6dbad67161333a1473d1e0b478b7475a0ec1 -> trunk/f7ad6dbad67161333a1473d1e0b478b7475a0ec1 2025-10-10T00:37:13.3799486Z * [new tag] trunk/fa5306b4f5bea89d80b9f14926712119aab78161 -> trunk/fa5306b4f5bea89d80b9f14926712119aab78161 2025-10-10T00:37:13.3799778Z * [new tag] trunk/fac6f20ae3a68fa49e19571a1fc4bcdddbf87d80 -> trunk/fac6f20ae3a68fa49e19571a1fc4bcdddbf87d80 2025-10-10T00:37:13.3800437Z * [new tag] trunk/fac85fcfb5ad0e63438d808a2f9ba7ea2dff9ad4 -> trunk/fac85fcfb5ad0e63438d808a2f9ba7ea2dff9ad4 2025-10-10T00:37:13.3800719Z * [new tag] trunk/fd3e15c14f4fc474af610b482382a2c85729f50d -> trunk/fd3e15c14f4fc474af610b482382a2c85729f50d 2025-10-10T00:37:13.3800983Z * [new tag] trunk/fd4bde430a51e5f216295c950d962c6343119821 -> trunk/fd4bde430a51e5f216295c950d962c6343119821 2025-10-10T00:37:13.3801238Z * [new tag] trunk/fdc622b513610b53ddcdc0b40282df9beae369bd -> trunk/fdc622b513610b53ddcdc0b40282df9beae369bd 2025-10-10T00:37:13.3801481Z * [new tag] trunk/fdc8ccc5bc433478c2a114016e193f5665d1e370 -> trunk/fdc8ccc5bc433478c2a114016e193f5665d1e370 2025-10-10T00:37:13.3801732Z * [new tag] trunk/ff5faa744a52561f4c6a138089123fd8d41cab73 -> trunk/ff5faa744a52561f4c6a138089123fd8d41cab73 2025-10-10T00:37:13.3801868Z * [new tag] v0.1.1 -> v0.1.1 2025-10-10T00:37:13.3801986Z * [new tag] v0.1.10 -> v0.1.10 2025-10-10T00:37:13.3802088Z * [new tag] v0.1.11 -> v0.1.11 2025-10-10T00:37:13.3802185Z * [new tag] v0.1.12 -> v0.1.12 2025-10-10T00:37:13.3802432Z * [new tag] v0.1.2 -> v0.1.2 2025-10-10T00:37:13.3802530Z * [new tag] v0.1.3 -> v0.1.3 2025-10-10T00:37:13.3802636Z * [new tag] v0.1.4 -> v0.1.4 2025-10-10T00:37:13.3804211Z * [new tag] v0.1.5 -> v0.1.5 2025-10-10T00:37:13.3804450Z * [new tag] v0.1.6 -> v0.1.6 2025-10-10T00:37:13.3804571Z * [new tag] v0.1.7 -> v0.1.7 2025-10-10T00:37:13.3804676Z * [new tag] v0.1.8 -> v0.1.8 2025-10-10T00:37:13.3804780Z * [new tag] v0.1.9 -> v0.1.9 2025-10-10T00:37:13.3804879Z * [new tag] v0.2.0 -> v0.2.0 2025-10-10T00:37:13.3804975Z * [new tag] v0.3.0 -> v0.3.0 2025-10-10T00:37:13.3805083Z * [new tag] v0.3.1 -> v0.3.1 2025-10-10T00:37:13.3805204Z * [new tag] v0.4.0 -> v0.4.0 2025-10-10T00:37:13.3805308Z * [new tag] v0.4.1 -> v0.4.1 2025-10-10T00:37:13.3809145Z * [new tag] v1.0.0 -> v1.0.0 2025-10-10T00:37:13.3809454Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-10-10T00:37:13.3809584Z * [new tag] v1.0.1 -> v1.0.1 2025-10-10T00:37:13.3809793Z * [new tag] v1.0rc0 -> v1.0rc0 2025-10-10T00:37:13.3810103Z * [new tag] v1.0rc1 -> v1.0rc1 2025-10-10T00:37:13.3810349Z * [new tag] v1.1.0 -> v1.1.0 2025-10-10T00:37:13.3810472Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-10-10T00:37:13.3811153Z * [new tag] v1.10.0 -> v1.10.0 2025-10-10T00:37:13.3811310Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-10-10T00:37:13.3811580Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-10-10T00:37:13.3811691Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-10-10T00:37:13.3811796Z * [new tag] v1.10.1 -> v1.10.1 2025-10-10T00:37:13.3811924Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-10-10T00:37:13.3812022Z * [new tag] v1.10.2 -> v1.10.2 2025-10-10T00:37:13.3812142Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-10-10T00:37:13.3812239Z * [new tag] v1.11.0 -> v1.11.0 2025-10-10T00:37:13.3812348Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-10-10T00:37:13.3812447Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-10-10T00:37:13.3812546Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-10-10T00:37:13.3818117Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-10-10T00:37:13.3818256Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-10-10T00:37:13.3818686Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-10-10T00:37:13.3818819Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-10-10T00:37:13.3818929Z * [new tag] v1.12.0 -> v1.12.0 2025-10-10T00:37:13.3819042Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-10-10T00:37:13.3819165Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-10-10T00:37:13.3819277Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-10-10T00:37:13.3819388Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-10-10T00:37:13.3819514Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-10-10T00:37:13.3819796Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-10-10T00:37:13.3819904Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-10-10T00:37:13.3820016Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-10-10T00:37:13.3820117Z * [new tag] v1.12.1 -> v1.12.1 2025-10-10T00:37:13.3820226Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-10-10T00:37:13.3822189Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-10-10T00:37:13.3822858Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-10-10T00:37:13.3823002Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-10-10T00:37:13.3823169Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-10-10T00:37:13.3823280Z * [new tag] v1.13.0 -> v1.13.0 2025-10-10T00:37:13.3823415Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-10-10T00:37:13.3823520Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-10-10T00:37:13.3823634Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-10-10T00:37:13.3823735Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-10-10T00:37:13.3824191Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-10-10T00:37:13.3824325Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-10-10T00:37:13.3825251Z * [new tag] v1.13.1 -> v1.13.1 2025-10-10T00:37:13.3825373Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-10-10T00:37:13.3825965Z * [new tag] v1.2.0 -> v1.2.0 2025-10-10T00:37:13.3826703Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-10-10T00:37:13.3827139Z * [new tag] v1.3.0 -> v1.3.0 2025-10-10T00:37:13.3830293Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-10-10T00:37:13.3830443Z * [new tag] v1.3.1 -> v1.3.1 2025-10-10T00:37:13.3830553Z * [new tag] v1.4.0 -> v1.4.0 2025-10-10T00:37:13.3830675Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-10-10T00:37:13.3830778Z * [new tag] v1.4.1 -> v1.4.1 2025-10-10T00:37:13.3831047Z * [new tag] v1.5.0 -> v1.5.0 2025-10-10T00:37:13.3832565Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-10-10T00:37:13.3832879Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-10-10T00:37:13.3833372Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-10-10T00:37:13.3833799Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-10-10T00:37:13.3834305Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-10-10T00:37:13.3835284Z * [new tag] v1.5.1 -> v1.5.1 2025-10-10T00:37:13.3835412Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-10-10T00:37:13.3835859Z * [new tag] v1.6.0 -> v1.6.0 2025-10-10T00:37:13.3838967Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-10-10T00:37:13.3839267Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-10-10T00:37:13.3839408Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-10-10T00:37:13.3839600Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-10-10T00:37:13.3839854Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-10-10T00:37:13.3839982Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-10-10T00:37:13.3840413Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-10-10T00:37:13.3840796Z * [new tag] v1.7.0 -> v1.7.0 2025-10-10T00:37:13.3840992Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-10-10T00:37:13.3842714Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-10-10T00:37:13.3843028Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-10-10T00:37:13.3843182Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-10-10T00:37:13.3843328Z * [new tag] v1.7.1 -> v1.7.1 2025-10-10T00:37:13.3845517Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-10-10T00:37:13.3845654Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-10-10T00:37:13.3845769Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-10-10T00:37:13.3845881Z * [new tag] v1.8.0 -> v1.8.0 2025-10-10T00:37:13.3846048Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-10-10T00:37:13.3847323Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-10-10T00:37:13.3847619Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-10-10T00:37:13.3847743Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-10-10T00:37:13.3848183Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-10-10T00:37:13.3848772Z * [new tag] v1.8.1 -> v1.8.1 2025-10-10T00:37:13.3849221Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-10-10T00:37:13.3849590Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-10-10T00:37:13.3850063Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-10-10T00:37:13.3854137Z * [new tag] v1.8.2 -> v1.8.2 2025-10-10T00:37:13.3854449Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-10-10T00:37:13.3854605Z * [new tag] v1.9.0 -> v1.9.0 2025-10-10T00:37:13.3854729Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-10-10T00:37:13.3854921Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-10-10T00:37:13.3855031Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-10-10T00:37:13.3855165Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-10-10T00:37:13.3855272Z * [new tag] v1.9.1 -> v1.9.1 2025-10-10T00:37:13.3855382Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-10-10T00:37:13.3856688Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-10-10T00:37:13.3856966Z * [new tag] v2.0.0 -> v2.0.0 2025-10-10T00:37:13.3857120Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-10-10T00:37:13.3859741Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-10-10T00:37:13.3860023Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-10-10T00:37:13.3860151Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-10-10T00:37:13.3860256Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-10-10T00:37:13.3860475Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-10-10T00:37:13.3860669Z * [new tag] v2.0.1 -> v2.0.1 2025-10-10T00:37:13.3861091Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-10-10T00:37:13.3861479Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-10-10T00:37:13.3862385Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-10-10T00:37:13.3863041Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-10-10T00:37:13.3863659Z * [new tag] v2.1.0 -> v2.1.0 2025-10-10T00:37:13.3864046Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-10-10T00:37:13.3864977Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-10-10T00:37:13.3865265Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-10-10T00:37:13.3866412Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-10-10T00:37:13.3866599Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-10-10T00:37:13.3867863Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-10-10T00:37:13.3868126Z * [new tag] v2.1.1 -> v2.1.1 2025-10-10T00:37:13.3870873Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-10-10T00:37:13.3871006Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-10-10T00:37:13.3871201Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-10-10T00:37:13.3871344Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-10-10T00:37:13.3871454Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-10-10T00:37:13.3877108Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-10-10T00:37:13.3877255Z * [new tag] v2.1.2 -> v2.1.2 2025-10-10T00:37:13.3877419Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-10-10T00:37:13.3877698Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-10-10T00:37:13.3877885Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-10-10T00:37:13.3878007Z * [new tag] v2.2.0 -> v2.2.0 2025-10-10T00:37:13.3878174Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-10-10T00:37:13.3878314Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-10-10T00:37:13.3878495Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-10-10T00:37:13.3878609Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-10-10T00:37:13.3884087Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-10-10T00:37:13.3884224Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-10-10T00:37:13.3884368Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-10-10T00:37:13.3884489Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-10-10T00:37:13.3884600Z * [new tag] v2.2.1 -> v2.2.1 2025-10-10T00:37:13.3884702Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-10-10T00:37:13.3884799Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-10-10T00:37:13.3884904Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-10-10T00:37:13.3885024Z * [new tag] v2.2.2 -> v2.2.2 2025-10-10T00:37:13.3885128Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-10-10T00:37:13.3885223Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-10-10T00:37:13.3885320Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-10-10T00:37:13.3885426Z * [new tag] v2.3.0 -> v2.3.0 2025-10-10T00:37:13.3885534Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-10-10T00:37:13.3885647Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-10-10T00:37:13.3885745Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-10-10T00:37:13.3885853Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-10-10T00:37:13.3885951Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-10-10T00:37:13.3886225Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-10-10T00:37:13.3886477Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-10-10T00:37:13.3886890Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-10-10T00:37:13.3888359Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-10-10T00:37:13.3888493Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-10-10T00:37:13.3888800Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-10-10T00:37:13.3888947Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-10-10T00:37:13.3889435Z * [new tag] v2.3.1 -> v2.3.1 2025-10-10T00:37:13.3893296Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-10-10T00:37:13.3893433Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-10-10T00:37:13.3893564Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-10-10T00:37:13.3893669Z * [new tag] v2.4.0 -> v2.4.0 2025-10-10T00:37:13.3893781Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-10-10T00:37:13.3893882Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-10-10T00:37:13.3893984Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-10-10T00:37:13.3894125Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-10-10T00:37:13.3895646Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-10-10T00:37:13.3895771Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-10-10T00:37:13.3896187Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-10-10T00:37:13.3896579Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-10-10T00:37:13.3901142Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-10-10T00:37:13.3901289Z * [new tag] v2.4.1 -> v2.4.1 2025-10-10T00:37:13.3901397Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-10-10T00:37:13.3901493Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-10-10T00:37:13.3901595Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-10-10T00:37:13.3901691Z * [new tag] v2.5.0 -> v2.5.0 2025-10-10T00:37:13.3901809Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-10-10T00:37:13.3901922Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-10-10T00:37:13.3902030Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-10-10T00:37:13.3903698Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-10-10T00:37:13.3903845Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-10-10T00:37:13.3903970Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-10-10T00:37:13.3904671Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-10-10T00:37:13.3905513Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-10-10T00:37:13.3905863Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-10-10T00:37:13.3907240Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-10-10T00:37:13.3907460Z * [new tag] v2.5.1 -> v2.5.1 2025-10-10T00:37:13.3907615Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-10-10T00:37:13.3907902Z * [new tag] v2.6.0 -> v2.6.0 2025-10-10T00:37:13.3910421Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-10-10T00:37:13.3910708Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-10-10T00:37:13.3911099Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-10-10T00:37:13.3911214Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-10-10T00:37:13.3911353Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-10-10T00:37:13.3912630Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-10-10T00:37:13.3912833Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-10-10T00:37:13.3915393Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-10-10T00:37:13.3915691Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-10-10T00:37:13.3915826Z * [new tag] v2.7.0 -> v2.7.0 2025-10-10T00:37:13.3915932Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-10-10T00:37:13.3916166Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-10-10T00:37:13.3929809Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-10-10T00:37:13.3929923Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-10-10T00:37:13.3930035Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-10-10T00:37:13.3930135Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-10-10T00:37:13.3930245Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-10-10T00:37:13.3930342Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-10-10T00:37:13.3930567Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-10-10T00:37:13.3930677Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-10-10T00:37:13.3930782Z * [new tag] v2.7.1 -> v2.7.1 2025-10-10T00:37:13.3930888Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-10-10T00:37:13.3930984Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-10-10T00:37:13.3931085Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-10-10T00:37:13.3931190Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-10-10T00:37:13.3931285Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-10-10T00:37:13.3931391Z * [new tag] v2.8.0 -> v2.8.0 2025-10-10T00:37:13.3931635Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-10-10T00:37:13.3931756Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-10-10T00:37:13.3931857Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-10-10T00:37:13.3931960Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-10-10T00:37:13.3932067Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-10-10T00:37:13.3932164Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-10-10T00:37:13.3932430Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-10-10T00:37:13.3932873Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-10-10T00:37:13.3933002Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-10-10T00:37:13.3933114Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-10-10T00:37:13.3933211Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-10-10T00:37:13.3933317Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-10-10T00:37:13.3933430Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-10-10T00:37:13.3936018Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-10-10T00:37:13.3942628Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-10-10T00:37:13.3942778Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-10-10T00:37:13.3943099Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-10-10T00:37:13.3943267Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-10-10T00:37:13.3943396Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-10-10T00:37:13.3943527Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-10-10T00:37:13.3943661Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-10-10T00:37:13.3943788Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-10-10T00:37:13.3943925Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-10-10T00:37:13.3944070Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-10-10T00:37:13.3944216Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-10-10T00:37:13.3944348Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-10-10T00:37:13.3944524Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-10-10T00:37:13.3944672Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-10-10T00:37:13.3944817Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-10-10T00:37:13.3944967Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-10-10T00:37:13.3945170Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-10-10T00:37:13.3945308Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-10-10T00:37:13.3945456Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-10-10T00:37:13.3945601Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-10-10T00:37:13.3945732Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-10-10T00:37:13.3945874Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-10-10T00:37:13.3946017Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-10-10T00:37:13.3946162Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-10-10T00:37:13.3946824Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-10-10T00:37:13.3951974Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-10-10T00:37:13.3953890Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-10-10T00:37:13.3954031Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-10-10T00:37:13.3954158Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-10-10T00:37:13.3954290Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-10-10T00:37:13.3954415Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-10-10T00:37:13.3954640Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-10-10T00:37:13.3960581Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-10-10T00:37:13.3964256Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-10-10T00:37:13.3966118Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-10-10T00:37:13.3966367Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-10-10T00:37:13.3972287Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-10-10T00:37:13.3977072Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-10-10T00:37:13.3982196Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-10-10T00:37:13.3982386Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-10-10T00:37:13.3982537Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-10-10T00:37:13.3982672Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-10-10T00:37:13.3982808Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-10-10T00:37:13.3982958Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-10-10T00:37:13.3983089Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-10-10T00:37:13.3983212Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-10-10T00:37:13.3983334Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-10-10T00:37:13.3983467Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-10-10T00:37:13.3983591Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-10-10T00:37:13.3983718Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-10-10T00:37:13.3983835Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-10-10T00:37:13.3983988Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-10-10T00:37:13.3984268Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-10-10T00:37:13.3984413Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-10-10T00:37:13.3984552Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-10-10T00:37:13.3984685Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-10-10T00:37:13.3984833Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-10-10T00:37:13.3984966Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-10-10T00:37:13.3985099Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-10-10T00:37:13.3985252Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-10-10T00:37:13.3985389Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-10-10T00:37:13.3985534Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-10-10T00:37:13.3985673Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-10-10T00:37:13.3985811Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-10-10T00:37:13.3985948Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-10-10T00:37:13.3986087Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-10-10T00:37:13.3986229Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-10-10T00:37:13.3986556Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-10-10T00:37:13.3986705Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-10-10T00:37:13.3986843Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-10-10T00:37:13.3986982Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-10-10T00:37:13.3987118Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-10-10T00:37:13.3987253Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-10-10T00:37:13.3987397Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-10-10T00:37:13.3987587Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-10-10T00:37:13.3987722Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-10-10T00:37:13.3987859Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-10-10T00:37:13.3987988Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-10-10T00:37:13.3988132Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-10-10T00:37:13.3988248Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-10-10T00:37:13.3988369Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-10-10T00:37:13.3988481Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-10-10T00:37:13.3988592Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-10-10T00:37:13.3988715Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-10-10T00:37:13.3988827Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-10-10T00:37:13.3988947Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-10-10T00:37:13.3989060Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-10-10T00:37:13.3989198Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-10-10T00:37:13.3989430Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-10-10T00:37:13.3989547Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-10-10T00:37:13.3989668Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-10-10T00:37:13.3989782Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-10-10T00:37:13.3989906Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-10-10T00:37:13.3990030Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-10-10T00:37:13.3990141Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-10-10T00:37:13.3990262Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-10-10T00:37:13.3990375Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-10-10T00:37:13.3990499Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-10-10T00:37:13.3990618Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-10-10T00:37:13.3990743Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-10-10T00:37:13.3990861Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-10-10T00:37:13.3990978Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-10-10T00:37:13.3991104Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-10-10T00:37:13.3991219Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-10-10T00:37:13.3991342Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-10-10T00:37:13.3991459Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-10-10T00:37:13.3991584Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-10-10T00:37:13.3991701Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-10-10T00:37:13.3991821Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-10-10T00:37:13.3991949Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-10-10T00:37:13.3992271Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-10-10T00:37:13.3992424Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-10-10T00:37:13.3992537Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-10-10T00:37:13.3992661Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-10-10T00:37:13.3992780Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-10-10T00:37:13.3992896Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-10-10T00:37:13.3993017Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-10-10T00:37:13.3993131Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-10-10T00:37:13.3993373Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-10-10T00:37:13.3993778Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-10-10T00:37:13.3995896Z * [new tag] whc_flight_1 -> whc_flight_1 2025-10-10T00:37:13.3996119Z * [new tag] whc_flight_2 -> whc_flight_2 2025-10-10T00:37:13.3996242Z * [new tag] whc_flight_4 -> whc_flight_4 2025-10-10T00:37:13.4523202Z [command]/usr/bin/git rev-parse --verify --quiet 344e6365a0068c2d2847fcec0c55dd53291d475e^{object} 2025-10-10T00:37:13.4555720Z 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:37:13.4563027Z ##[endgroup] 2025-10-10T00:37:13.4563354Z ##[group]Determining the checkout info 2025-10-10T00:37:13.4563687Z ##[endgroup] 2025-10-10T00:37:13.4569672Z [command]/usr/bin/git sparse-checkout disable 2025-10-10T00:37:13.4607394Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-10-10T00:37:13.4661385Z ##[group]Checking out the ref 2025-10-10T00:37:13.4661798Z [command]/usr/bin/git checkout --progress --force 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:37:14.4913156Z Updating files: 97% (19420/19929) 2025-10-10T00:37:14.5138087Z Updating files: 98% (19531/19929) 2025-10-10T00:37:14.5272617Z Updating files: 99% (19730/19929) 2025-10-10T00:37:14.5272882Z Updating files: 100% (19929/19929) 2025-10-10T00:37:14.5273091Z Updating files: 100% (19929/19929), done. 2025-10-10T00:37:14.5497479Z Note: switching to '344e6365a0068c2d2847fcec0c55dd53291d475e'. 2025-10-10T00:37:14.5498004Z 2025-10-10T00:37:14.5498389Z You are in 'detached HEAD' state. You can look around, make experimental 2025-10-10T00:37:14.5498786Z changes and commit them, and you can discard any commits you make in this 2025-10-10T00:37:14.5499182Z state without impacting any branches by switching back to a branch. 2025-10-10T00:37:14.5499437Z 2025-10-10T00:37:14.5499590Z If you want to create a new branch to retain commits you create, you may 2025-10-10T00:37:14.5499955Z do so (now or later) by using -c with the switch command. Example: 2025-10-10T00:37:14.5500157Z 2025-10-10T00:37:14.5500253Z git switch -c 2025-10-10T00:37:14.5500403Z 2025-10-10T00:37:14.5500617Z Or undo this operation with: 2025-10-10T00:37:14.5500767Z 2025-10-10T00:37:14.5500844Z git switch - 2025-10-10T00:37:14.5500951Z 2025-10-10T00:37:14.5501126Z Turn off this advice by setting config variable advice.detachedHead to false 2025-10-10T00:37:14.5501365Z 2025-10-10T00:37:14.5501611Z HEAD is now at 344e6365a00 [inductor][eazy] change how torch.use_deterministic_algorithms affect inductor (#164905) 2025-10-10T00:37:14.5548622Z ##[endgroup] 2025-10-10T00:37:14.5548978Z ##[group]Setting up auth for fetching submodules 2025-10-10T00:37:14.5555733Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-10-10T00:37:14.5613525Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-10-10T00:37:14.5653531Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-10-10T00:37:14.5692613Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-10-10T00:37:14.5716591Z ##[endgroup] 2025-10-10T00:37:14.5717112Z ##[group]Fetching submodules 2025-10-10T00:37:14.5722904Z [command]/usr/bin/git submodule sync --recursive 2025-10-10T00:37:14.6082657Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-10-10T00:37:14.6536319Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-10-10T00:37:14.6537058Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-10-10T00:37:14.6537710Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-10-10T00:37:14.6538353Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-10-10T00:37:14.6538951Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-10-10T00:37:14.6539708Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-10-10T00:37:14.6540469Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-10-10T00:37:14.6541260Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-10-10T00:37:14.6541831Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-10-10T00:37:14.6542504Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-10-10T00:37:14.6543203Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-10-10T00:37:14.6543817Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-10-10T00:37:14.6544456Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-10-10T00:37:14.6545090Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-10-10T00:37:14.6545660Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-10-10T00:37:14.6546533Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-10-10T00:37:14.6547253Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-10-10T00:37:14.6547842Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-10-10T00:37:14.6548503Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:37:14.6549091Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-10-10T00:37:14.6549650Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-10-10T00:37:14.6550207Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-10-10T00:37:14.6550729Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-10-10T00:37:14.6551256Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-10-10T00:37:14.6551920Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-10-10T00:37:14.6552515Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-10-10T00:37:14.6553099Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-10-10T00:37:14.6553587Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-10-10T00:37:14.6554166Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-10-10T00:37:14.6554807Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-10-10T00:37:14.6555385Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-10-10T00:37:14.6555981Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-10-10T00:37:14.6556616Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-10-10T00:37:14.6557219Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-10-10T00:37:14.6557827Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-10-10T00:37:14.6558373Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-10-10T00:37:14.6558903Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-10-10T00:37:14.6570871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-10-10T00:37:14.8796647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-10-10T00:37:14.8797359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-10-10T00:37:14.8798213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-10-10T00:37:14.8798693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-10-10T00:37:14.8821811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-10-10T00:37:15.1929851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-10-10T00:37:15.1930378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-10-10T00:37:15.1930850Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-10-10T00:37:15.1931348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-10-10T00:37:15.1931996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-10-10T00:37:15.1932505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-10-10T00:37:15.1932959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-10-10T00:37:15.1933581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-10-10T00:37:15.2001525Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-10-10T00:37:16.7414527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-10-10T00:37:16.7415299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-10-10T00:37:16.7416117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-10-10T00:37:16.7417099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-10-10T00:37:16.7417665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-10-10T00:37:16.7418123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-10-10T00:37:16.7418645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-10-10T00:37:16.7419303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-10-10T00:37:16.7419776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-10-10T00:37:16.7420253Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-10-10T00:37:16.7420732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-10-10T00:37:16.7421473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-10-10T00:37:16.8416109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-10-10T00:37:28.3977668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-10-10T00:37:28.3982343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-10-10T00:37:28.3982885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-10-10T00:37:28.3983679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-10-10T00:37:28.3984180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-10-10T00:37:28.3984661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-10-10T00:37:28.3985144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-10-10T00:37:28.3985642Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-10-10T00:37:28.3986105Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-10-10T00:37:28.4124378Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-10-10T00:37:28.4247332Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-10-10T00:37:28.4328362Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-10-10T00:37:28.4533851Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-10-10T00:37:28.5206719Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-10-10T00:37:28.5683916Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-10-10T00:37:29.1164825Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-10-10T00:37:29.2540555Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-10-10T00:37:29.2558516Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:37:29.2584612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-10-10T00:37:33.2949052Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-10-10T00:37:33.3163641Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-10-10T00:37:33.5742932Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-10-10T00:37:33.6144918Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-10-10T00:37:33.7038598Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-10-10T00:37:33.7434439Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-10-10T00:37:34.2607648Z Submodule path 'third_party/cutlass': checked out 'f3fde58372d33e9a5650ba7b80fc48b3b49d40c8' 2025-10-10T00:37:34.3838013Z Submodule path 'third_party/fbgemm': checked out '3cefe0564a8c3de514a152d40a2b4770f2ee5be0' 2025-10-10T00:37:34.3853464Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-10-10T00:37:34.3855527Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:37:34.3856265Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:37:34.3856929Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-10-10T00:37:34.3861345Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-10-10T00:37:34.3862353Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:37:34.3863451Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-10-10T00:37:34.3892781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-10-10T00:37:35.5602151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-10-10T00:37:35.5602820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-10-10T00:37:35.5603362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-10-10T00:37:35.6604208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-10-10T00:37:38.1294261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-10-10T00:37:38.2292310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-10-10T00:37:41.0839695Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-10-10T00:37:41.3434320Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-10-10T00:37:41.4320037Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-10-10T00:37:41.9604805Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '311f3c8e51dc0eb56310cfc6980bf63d0fbd7917' 2025-10-10T00:37:42.0018995Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T00:37:42.0145719Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-10-10T00:37:42.1105142Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-10-10T00:37:42.1697614Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-10-10T00:37:42.1715211Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:37:42.1716013Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:37:42.1744847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-10-10T00:37:45.9676982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-10-10T00:37:46.1552692Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-10-10T00:37:46.6366787Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-10-10T00:37:46.7516730Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-10-10T00:37:46.7796136Z Submodule path 'third_party/fmt': checked out 'e424e3f2e607da02742f73db84873b8084fc714c' 2025-10-10T00:37:46.8143464Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-10-10T00:37:46.8363638Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-10-10T00:37:46.8760695Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T00:37:46.8889304Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-10-10T00:37:46.8902840Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-10-10T00:37:46.8927881Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-10-10T00:38:01.8957161Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-10-10T00:38:01.9128899Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-10-10T00:38:01.9956438Z Submodule path 'third_party/kineto': checked out '001ba8eb519438592f79dbc8e86a349f5f6c6829' 2025-10-10T00:38:01.9977937Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:01.9978782Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:01.9979543Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:02.0004222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-10-10T00:38:02.6548528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-10-10T00:38:03.2088965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-10-10T00:38:03.2857698Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-10-10T00:38:03.2877534Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:03.2878543Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:03.2879264Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:03.2880072Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:03.2886478Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:03.2888348Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:03.2889118Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:03.2889865Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:03.2890699Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:03.2918481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-10-10T00:38:04.6032267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-10-10T00:38:04.6033102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-10-10T00:38:04.6033849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-10-10T00:38:04.6034538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-10-10T00:38:04.6035494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-10-10T00:38:04.6036210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-10-10T00:38:04.6036897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-10-10T00:38:04.7036358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-10-10T00:38:10.0523144Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-10-10T00:38:10.0678791Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-10-10T00:38:10.0999565Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-10-10T00:38:10.1114658Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-10-10T00:38:10.1131297Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:10.1159873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-10-10T00:38:10.3807637Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-10-10T00:38:10.3981901Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-10-10T00:38:10.4380667Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T00:38:10.5232145Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-10-10T00:38:10.5387485Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-10-10T00:38:10.5527556Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-10-10T00:38:10.5545667Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:10.5546813Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:10.5572918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-10-10T00:38:12.3966580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-10-10T00:38:12.6123978Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-10-10T00:38:12.6528176Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-10-10T00:38:12.6827285Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-10-10T00:38:12.7228635Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-10-10T00:38:12.7613537Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-10-10T00:38:12.7969897Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-10-10T00:38:12.8956426Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-10-10T00:38:13.1886069Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-10-10T00:38:13.1916674Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:13.1947234Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-10-10T00:38:14.2223357Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-10-10T00:38:14.2779250Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-10-10T00:38:14.2796927Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:14.2798799Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:14.2799518Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:14.2800188Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:14.2800984Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:14.2801876Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:14.2803701Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:14.2804737Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:14.2831407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-10-10T00:38:14.6956933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-10-10T00:38:14.6957688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-10-10T00:38:14.6958401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-10-10T00:38:14.6959038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-10-10T00:38:14.7962172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-10-10T00:38:15.3597668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-10-10T00:38:20.9681883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-10-10T00:38:21.6783169Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-10-10T00:38:21.7145180Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-10-10T00:38:21.7306935Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-10-10T00:38:21.8207124Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-10-10T00:38:21.8339299Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-10-10T00:38:21.8472768Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-10-10T00:38:21.8616990Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-10-10T00:38:21.8629443Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:21.8630250Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:21.8661372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-10-10T00:38:23.6697422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-10-10T00:38:23.8943439Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-10-10T00:38:23.9356066Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-10-10T00:38:24.2663792Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-10-10T00:38:24.2777307Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-10-10T00:38:24.5017107Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-10-10T00:38:24.5044506Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:24.5045846Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:24.5073727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-10-10T00:38:25.0053286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-10-10T00:38:25.5081097Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-10-10T00:38:25.5755894Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-10-10T00:38:25.5854333Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-10-10T00:38:25.5972928Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-10-10T00:38:25.6326975Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-10-10T00:38:25.6591133Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-10-10T00:38:25.6994239Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-10-10T00:38:25.7233204Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075' 2025-10-10T00:38:25.7255117Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:25.7259824Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:25.7261954Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:25.7262631Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:25.7289316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-10-10T00:38:26.7076744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-10-10T00:38:26.7567309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-10-10T00:38:26.9388527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-10-10T00:38:26.9915710Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-10-10T00:38:27.0064156Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-10-10T00:38:27.0712241Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-10-10T00:38:27.0981901Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-10-10T00:38:27.1000059Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:27.1022255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-10-10T00:38:27.3047608Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-10-10T00:38:27.3091071Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-10-10T00:38:27.3427370Z Entering 'android/libs/fbjni' 2025-10-10T00:38:27.3471514Z Entering 'third_party/FP16' 2025-10-10T00:38:27.3509672Z Entering 'third_party/FXdiv' 2025-10-10T00:38:27.3550318Z Entering 'third_party/NNPACK' 2025-10-10T00:38:27.3587035Z Entering 'third_party/NVTX' 2025-10-10T00:38:27.3630050Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:27.3666596Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:27.3721056Z Entering 'third_party/aiter' 2025-10-10T00:38:27.3762823Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:27.3810564Z Entering 'third_party/benchmark' 2025-10-10T00:38:27.3858053Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:27.3904762Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:27.3946604Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:27.3991908Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:27.4028388Z Entering 'third_party/cutlass' 2025-10-10T00:38:27.4083577Z Entering 'third_party/fbgemm' 2025-10-10T00:38:27.4122599Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:27.4168529Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:27.4207904Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:27.4251330Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:27.4302270Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:27.4345394Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:27.4384567Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:27.4429463Z Entering 'third_party/flash-attention' 2025-10-10T00:38:27.4473010Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:27.4512123Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:27.4564955Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:27.4600022Z Entering 'third_party/fmt' 2025-10-10T00:38:27.4643690Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:27.4684058Z Entering 'third_party/gloo' 2025-10-10T00:38:27.4727716Z Entering 'third_party/googletest' 2025-10-10T00:38:27.4768535Z Entering 'third_party/ideep' 2025-10-10T00:38:27.4808007Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:27.4861316Z Entering 'third_party/ittapi' 2025-10-10T00:38:27.4901167Z Entering 'third_party/kineto' 2025-10-10T00:38:27.4941186Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:27.4983491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:27.5024298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:27.5069578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:27.5105037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:27.5145878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:27.5188255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:27.5228591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:27.5273770Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:27.5309410Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:27.5351817Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:27.5393118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:27.5425607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:27.5477783Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:27.5516164Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:27.5567094Z Entering 'third_party/kleidiai' 2025-10-10T00:38:27.5604043Z Entering 'third_party/mimalloc' 2025-10-10T00:38:27.5647557Z Entering 'third_party/nlohmann' 2025-10-10T00:38:27.5689262Z Entering 'third_party/onnx' 2025-10-10T00:38:27.5739218Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:27.5784973Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:27.5825189Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:27.5865266Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:27.5904366Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:27.5946136Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:27.5983914Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:27.6027081Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:27.6065495Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:27.6101939Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:27.6146767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:27.6194229Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:27.6258799Z Entering 'third_party/pocketfft' 2025-10-10T00:38:27.6294221Z Entering 'third_party/protobuf' 2025-10-10T00:38:27.6340024Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:27.6377998Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:27.6417479Z Entering 'third_party/psimd' 2025-10-10T00:38:27.6464214Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:27.6503500Z Entering 'third_party/pybind11' 2025-10-10T00:38:27.6546761Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:27.6585035Z Entering 'third_party/sleef' 2025-10-10T00:38:27.6624584Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:27.6667291Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:27.6703324Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:27.6744762Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:27.6785008Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:27.6822115Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:27.6885996Z ##[endgroup] 2025-10-10T00:38:27.6886498Z ##[group]Persisting credentials for submodules 2025-10-10T00:38:27.6887170Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-10-10T00:38:27.7197017Z Entering 'android/libs/fbjni' 2025-10-10T00:38:27.7250049Z Entering 'third_party/FP16' 2025-10-10T00:38:27.7300046Z Entering 'third_party/FXdiv' 2025-10-10T00:38:27.7359807Z Entering 'third_party/NNPACK' 2025-10-10T00:38:27.7410102Z Entering 'third_party/NVTX' 2025-10-10T00:38:27.7466016Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:27.7516431Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:27.7588431Z Entering 'third_party/aiter' 2025-10-10T00:38:27.7642462Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:27.7704298Z Entering 'third_party/benchmark' 2025-10-10T00:38:27.7761927Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:27.7821581Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:27.7884443Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:27.7937836Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:27.7992679Z Entering 'third_party/cutlass' 2025-10-10T00:38:27.8057655Z Entering 'third_party/fbgemm' 2025-10-10T00:38:27.8111877Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:27.8171895Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:27.8236569Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:27.8289132Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:27.8346957Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:27.8403407Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:27.8459490Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:27.8519925Z Entering 'third_party/flash-attention' 2025-10-10T00:38:27.8579150Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:27.8647458Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:27.8703287Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:27.8766517Z Entering 'third_party/fmt' 2025-10-10T00:38:27.8819193Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:27.8875835Z Entering 'third_party/gloo' 2025-10-10T00:38:27.8925857Z Entering 'third_party/googletest' 2025-10-10T00:38:27.8983720Z Entering 'third_party/ideep' 2025-10-10T00:38:27.9039815Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:27.9097055Z Entering 'third_party/ittapi' 2025-10-10T00:38:27.9160604Z Entering 'third_party/kineto' 2025-10-10T00:38:27.9212964Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:27.9265934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:27.9322155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:27.9379329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:27.9431760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:27.9481750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:27.9536677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:27.9597540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:27.9658668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:27.9706121Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:27.9763120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:27.9813300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:27.9874833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:27.9936427Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:27.9990775Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:28.0056951Z Entering 'third_party/kleidiai' 2025-10-10T00:38:28.0109685Z Entering 'third_party/mimalloc' 2025-10-10T00:38:28.0167036Z Entering 'third_party/nlohmann' 2025-10-10T00:38:28.0218410Z Entering 'third_party/onnx' 2025-10-10T00:38:28.0286146Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:28.0346391Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:28.0401243Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:28.0461197Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:28.0513187Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:28.0572493Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:28.0622777Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:28.0682395Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:28.0737559Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:28.0787587Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:28.0840122Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:28.0897389Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:28.0981577Z Entering 'third_party/pocketfft' 2025-10-10T00:38:28.1028501Z Entering 'third_party/protobuf' 2025-10-10T00:38:28.1087157Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:28.1145065Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:28.1204692Z Entering 'third_party/psimd' 2025-10-10T00:38:28.1260959Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:28.1313395Z Entering 'third_party/pybind11' 2025-10-10T00:38:28.1368240Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:28.1421678Z Entering 'third_party/sleef' 2025-10-10T00:38:28.1485676Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:28.1532195Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:28.1585488Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:28.1647710Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:28.1696907Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:28.1747057Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:28.1818608Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-10-10T00:38:28.2133462Z Entering 'android/libs/fbjni' 2025-10-10T00:38:28.2192761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-10-10T00:38:28.2204236Z Entering 'third_party/FP16' 2025-10-10T00:38:28.2256564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-10-10T00:38:28.2277399Z Entering 'third_party/FXdiv' 2025-10-10T00:38:28.2324346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-10-10T00:38:28.2342444Z Entering 'third_party/NNPACK' 2025-10-10T00:38:28.2398704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-10-10T00:38:28.2411499Z Entering 'third_party/NVTX' 2025-10-10T00:38:28.2463651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-10-10T00:38:28.2484463Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:28.2533830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-10-10T00:38:28.2552268Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:28.2600564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-10-10T00:38:28.2627402Z Entering 'third_party/aiter' 2025-10-10T00:38:28.2678653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-10-10T00:38:28.2695470Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:28.2745980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-10-10T00:38:28.2773409Z Entering 'third_party/benchmark' 2025-10-10T00:38:28.2822957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-10-10T00:38:28.2842086Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:28.2890040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-10-10T00:38:28.2911785Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:28.2963417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-10-10T00:38:28.2981408Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:28.3028692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-10-10T00:38:28.3052009Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:28.3094722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-10-10T00:38:28.3111698Z Entering 'third_party/cutlass' 2025-10-10T00:38:28.3167509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-10-10T00:38:28.3193780Z Entering 'third_party/fbgemm' 2025-10-10T00:38:28.3244054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-10-10T00:38:28.3261612Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:28.3305878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-10-10T00:38:28.3322454Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:28.3370527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-10-10T00:38:28.3395039Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:28.3446820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-10-10T00:38:28.3462484Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:28.3515708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-10-10T00:38:28.3544399Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:28.3590999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-10-10T00:38:28.3603926Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:28.3657836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-10-10T00:38:28.3674713Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:28.3721854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-10-10T00:38:28.3744951Z Entering 'third_party/flash-attention' 2025-10-10T00:38:28.3795232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-10-10T00:38:28.3808084Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:28.3860061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-10-10T00:38:28.3881323Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:28.3928190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-10-10T00:38:28.3957246Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:28.4007267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-10-10T00:38:28.4022987Z Entering 'third_party/fmt' 2025-10-10T00:38:28.4074880Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-10-10T00:38:28.4092311Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:28.4139617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-10-10T00:38:28.4166220Z Entering 'third_party/gloo' 2025-10-10T00:38:28.4211961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-10-10T00:38:28.4225569Z Entering 'third_party/googletest' 2025-10-10T00:38:28.4277721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.4295593Z Entering 'third_party/ideep' 2025-10-10T00:38:28.4343962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-10-10T00:38:28.4361991Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:28.4408163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-10-10T00:38:28.4432011Z Entering 'third_party/ittapi' 2025-10-10T00:38:28.4482178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-10-10T00:38:28.4494753Z Entering 'third_party/kineto' 2025-10-10T00:38:28.4543120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-10-10T00:38:28.4562915Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:28.4609452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-10-10T00:38:28.4624370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:28.4677552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-10-10T00:38:28.4695515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:28.4746022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-10-10T00:38:28.4768004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:28.4811226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-10-10T00:38:28.4828645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:28.4885801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-10-10T00:38:28.4900890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:28.4954744Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-10-10T00:38:28.4970445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:28.5017233Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-10-10T00:38:28.5033257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:28.5087800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.5104819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:28.5158326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-10-10T00:38:28.5175155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:28.5221392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-10-10T00:38:28.5242525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:28.5289068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-10-10T00:38:28.5303754Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:28.5363241Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-10-10T00:38:28.5381784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:28.5431101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-10-10T00:38:28.5455852Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:28.5508799Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-10-10T00:38:28.5522652Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:28.5574360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.5591445Z Entering 'third_party/kleidiai' 2025-10-10T00:38:28.5643846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-10-10T00:38:28.5662263Z Entering 'third_party/mimalloc' 2025-10-10T00:38:28.5711063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-10-10T00:38:28.5724276Z Entering 'third_party/nlohmann' 2025-10-10T00:38:28.5774682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-10-10T00:38:28.5792737Z Entering 'third_party/onnx' 2025-10-10T00:38:28.5842571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-10-10T00:38:28.5875489Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:28.5919780Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-10-10T00:38:28.5946511Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:28.5997881Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-10-10T00:38:28.6011695Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:28.6066615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-10-10T00:38:28.6083749Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:28.6129572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.6151866Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:28.6192230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-10-10T00:38:28.6207083Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:28.6256023Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-10-10T00:38:28.6271829Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:28.6323205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-10-10T00:38:28.6340605Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:28.6391246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-10-10T00:38:28.6407921Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:28.6460488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-10-10T00:38:28.6477092Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:28.6525582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-10-10T00:38:28.6543196Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:28.6592566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-10-10T00:38:28.6607910Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:28.6660522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-10-10T00:38:28.6690797Z Entering 'third_party/pocketfft' 2025-10-10T00:38:28.6743305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-10-10T00:38:28.6759542Z Entering 'third_party/protobuf' 2025-10-10T00:38:28.6805405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-10-10T00:38:28.6822936Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:28.6877966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-10-10T00:38:28.6900146Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:28.6945094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.6967551Z Entering 'third_party/psimd' 2025-10-10T00:38:28.7011888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-10-10T00:38:28.7031857Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:28.7080768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-10-10T00:38:28.7090047Z Entering 'third_party/pybind11' 2025-10-10T00:38:28.7144527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-10-10T00:38:28.7163659Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:28.7212182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-10-10T00:38:28.7224396Z Entering 'third_party/sleef' 2025-10-10T00:38:28.7280966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-10-10T00:38:28.7298519Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:28.7347041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-10-10T00:38:28.7360077Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:28.7413183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-10-10T00:38:28.7430468Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:28.7478980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-10-10T00:38:28.7492547Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:28.7539546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-10-10T00:38:28.7559619Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:28.7605742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-10-10T00:38:28.7621065Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:28.7674588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-10-10T00:38:28.9334834Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-10-10T00:38:28.9665602Z Entering 'android/libs/fbjni' 2025-10-10T00:38:28.9703997Z Entering 'third_party/FP16' 2025-10-10T00:38:28.9748254Z Entering 'third_party/FXdiv' 2025-10-10T00:38:28.9794589Z Entering 'third_party/NNPACK' 2025-10-10T00:38:28.9828174Z Entering 'third_party/NVTX' 2025-10-10T00:38:28.9872919Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:28.9910185Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:28.9970421Z Entering 'third_party/aiter' 2025-10-10T00:38:29.0006542Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:29.0053506Z Entering 'third_party/benchmark' 2025-10-10T00:38:29.0095990Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:29.0147037Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:29.0193259Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:29.0227370Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:29.0271794Z Entering 'third_party/cutlass' 2025-10-10T00:38:29.0312086Z Entering 'third_party/fbgemm' 2025-10-10T00:38:29.0355233Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:29.0397215Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:29.0442898Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:29.0483644Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:29.0525441Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:29.0572223Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:29.0611169Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:29.0660435Z Entering 'third_party/flash-attention' 2025-10-10T00:38:29.0698788Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:29.0745013Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:29.0792423Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:29.0831120Z Entering 'third_party/fmt' 2025-10-10T00:38:29.0879897Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:29.0914248Z Entering 'third_party/gloo' 2025-10-10T00:38:29.0960693Z Entering 'third_party/googletest' 2025-10-10T00:38:29.0999912Z Entering 'third_party/ideep' 2025-10-10T00:38:29.1040997Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:29.1088521Z Entering 'third_party/ittapi' 2025-10-10T00:38:29.1123042Z Entering 'third_party/kineto' 2025-10-10T00:38:29.1171466Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:29.1213204Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:29.1256550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:29.1299885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:29.1340791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:29.1377267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:29.1415373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:29.1458840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:29.1502941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:29.1546477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:29.1587191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:29.1627072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:29.1671628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:29.1710601Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:29.1754055Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:29.1796351Z Entering 'third_party/kleidiai' 2025-10-10T00:38:29.1841646Z Entering 'third_party/mimalloc' 2025-10-10T00:38:29.1888154Z Entering 'third_party/nlohmann' 2025-10-10T00:38:29.1925168Z Entering 'third_party/onnx' 2025-10-10T00:38:29.1981375Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:29.2024614Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:29.2072402Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:29.2111009Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:29.2156134Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:29.2192857Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:29.2231082Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:29.2276151Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:29.2316249Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:29.2356077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:29.2401817Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:29.2444210Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:29.2498260Z Entering 'third_party/pocketfft' 2025-10-10T00:38:29.2542846Z Entering 'third_party/protobuf' 2025-10-10T00:38:29.2582852Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:29.2621259Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:29.2670394Z Entering 'third_party/psimd' 2025-10-10T00:38:29.2708644Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:29.2747847Z Entering 'third_party/pybind11' 2025-10-10T00:38:29.2782613Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:29.2824185Z Entering 'third_party/sleef' 2025-10-10T00:38:29.2865619Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:29.2904175Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:29.2947491Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:29.2993253Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:29.3029693Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:29.3071806Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:29.3126674Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-10-10T00:38:29.3441218Z Entering 'android/libs/fbjni' 2025-10-10T00:38:29.3481447Z Entering 'third_party/FP16' 2025-10-10T00:38:29.3525160Z Entering 'third_party/FXdiv' 2025-10-10T00:38:29.3566817Z Entering 'third_party/NNPACK' 2025-10-10T00:38:29.3608477Z Entering 'third_party/NVTX' 2025-10-10T00:38:29.3656201Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:29.3692620Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:29.3744776Z Entering 'third_party/aiter' 2025-10-10T00:38:29.3784222Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:29.3836584Z Entering 'third_party/benchmark' 2025-10-10T00:38:29.3877236Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:29.3920581Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:29.3963087Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:29.4000391Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:29.4047597Z Entering 'third_party/cutlass' 2025-10-10T00:38:29.4096126Z Entering 'third_party/fbgemm' 2025-10-10T00:38:29.4137014Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:29.4177356Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:29.4220691Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:29.4263479Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:29.4313910Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:29.4359277Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:29.4399841Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:29.4444261Z Entering 'third_party/flash-attention' 2025-10-10T00:38:29.4488763Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:29.4537169Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:29.4591930Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:29.4630558Z Entering 'third_party/fmt' 2025-10-10T00:38:29.4678214Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:29.4720937Z Entering 'third_party/gloo' 2025-10-10T00:38:29.4768958Z Entering 'third_party/googletest' 2025-10-10T00:38:29.4813112Z Entering 'third_party/ideep' 2025-10-10T00:38:29.4856019Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:29.4897700Z Entering 'third_party/ittapi' 2025-10-10T00:38:29.4940666Z Entering 'third_party/kineto' 2025-10-10T00:38:29.4981820Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:29.5021123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:29.5066097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:29.5107240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:29.5150805Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:29.5190033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:29.5232981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:29.5276903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:29.5315921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:29.5363160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:29.5400081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:29.5443833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:29.5482852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:29.5522683Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:29.5565155Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:29.5606936Z Entering 'third_party/kleidiai' 2025-10-10T00:38:29.5647319Z Entering 'third_party/mimalloc' 2025-10-10T00:38:29.5684913Z Entering 'third_party/nlohmann' 2025-10-10T00:38:29.5725843Z Entering 'third_party/onnx' 2025-10-10T00:38:29.5777221Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:29.5818362Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:29.5864848Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:29.5904844Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:29.5950949Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:29.5994078Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:29.6027206Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:29.6065911Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:29.6104637Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:29.6143286Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:29.6185491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:29.6227589Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:29.6284864Z Entering 'third_party/pocketfft' 2025-10-10T00:38:29.6324572Z Entering 'third_party/protobuf' 2025-10-10T00:38:29.6374380Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:29.6415763Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:29.6463499Z Entering 'third_party/psimd' 2025-10-10T00:38:29.6503502Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:29.6542598Z Entering 'third_party/pybind11' 2025-10-10T00:38:29.6583971Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:29.6625500Z Entering 'third_party/sleef' 2025-10-10T00:38:29.6664447Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:29.6702541Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:29.6745153Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:29.6784630Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:29.6821143Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:29.6864384Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:29.6920381Z ##[endgroup] 2025-10-10T00:38:29.6964849Z [command]/usr/bin/git log -1 --format=%H 2025-10-10T00:38:29.6995089Z 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:38:29.7102115Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-10-10T00:38:29.7102371Z cd "${GITHUB_WORKSPACE}" 2025-10-10T00:38:29.7102579Z # Clean stale submodule dirs 2025-10-10T00:38:29.7102796Z if [ -z "${NO_SUDO}" ]; then 2025-10-10T00:38:29.7103051Z  sudo git submodule foreach --recursive git clean -ffdx 2025-10-10T00:38:29.7103297Z else 2025-10-10T00:38:29.7103509Z  git submodule foreach --recursive git clean -ffdx 2025-10-10T00:38:29.7103742Z fi 2025-10-10T00:38:29.7112313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:29.7112571Z env: 2025-10-10T00:38:29.7112744Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:29.7112929Z NO_SUDO: true 2025-10-10T00:38:29.7113089Z ##[endgroup] 2025-10-10T00:38:29.7453083Z Entering 'android/libs/fbjni' 2025-10-10T00:38:29.7486359Z Entering 'third_party/FP16' 2025-10-10T00:38:29.7515090Z Entering 'third_party/FXdiv' 2025-10-10T00:38:29.7542057Z Entering 'third_party/NNPACK' 2025-10-10T00:38:29.7572448Z Entering 'third_party/NVTX' 2025-10-10T00:38:29.7611075Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T00:38:29.7647470Z Entering 'third_party/XNNPACK' 2025-10-10T00:38:29.7741610Z Entering 'third_party/aiter' 2025-10-10T00:38:29.7779218Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T00:38:29.7870048Z Entering 'third_party/benchmark' 2025-10-10T00:38:29.7899863Z Entering 'third_party/composable_kernel' 2025-10-10T00:38:29.7994916Z Entering 'third_party/cpp-httplib' 2025-10-10T00:38:29.8025892Z Entering 'third_party/cpuinfo' 2025-10-10T00:38:29.8063774Z Entering 'third_party/cudnn_frontend' 2025-10-10T00:38:29.8099722Z Entering 'third_party/cutlass' 2025-10-10T00:38:29.8181704Z Entering 'third_party/fbgemm' 2025-10-10T00:38:29.8232363Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T00:38:29.8262054Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T00:38:29.8363987Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T00:38:29.8396474Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T00:38:29.8489532Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T00:38:29.8519021Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T00:38:29.8553043Z Entering 'third_party/fbgemm/external/json' 2025-10-10T00:38:29.8596748Z Entering 'third_party/flash-attention' 2025-10-10T00:38:29.8628830Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T00:38:29.8704027Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T00:38:29.8783397Z Entering 'third_party/flatbuffers' 2025-10-10T00:38:29.8844918Z Entering 'third_party/fmt' 2025-10-10T00:38:29.8878247Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T00:38:29.8907160Z Entering 'third_party/gloo' 2025-10-10T00:38:29.8938935Z Entering 'third_party/googletest' 2025-10-10T00:38:29.8968069Z Entering 'third_party/ideep' 2025-10-10T00:38:29.8998318Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T00:38:29.9075097Z Entering 'third_party/ittapi' 2025-10-10T00:38:29.9108304Z Entering 'third_party/kineto' 2025-10-10T00:38:29.9142391Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T00:38:29.9179287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T00:38:29.9215587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T00:38:29.9249999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T00:38:29.9285542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T00:38:29.9307700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T00:38:29.9341599Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T00:38:29.9374681Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T00:38:29.9402555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T00:38:29.9438900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T00:38:29.9469283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T00:38:29.9503588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:29.9547014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:29.9582479Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T00:38:29.9615664Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T00:38:29.9651254Z Entering 'third_party/kleidiai' 2025-10-10T00:38:29.9689388Z Entering 'third_party/mimalloc' 2025-10-10T00:38:29.9721317Z Entering 'third_party/nlohmann' 2025-10-10T00:38:29.9763102Z Entering 'third_party/onnx' 2025-10-10T00:38:29.9987596Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T00:38:30.0021961Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T00:38:30.0070835Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T00:38:30.0102625Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T00:38:30.0131886Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T00:38:30.0162643Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T00:38:30.0203752Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T00:38:30.0233200Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T00:38:30.0263182Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T00:38:30.0294573Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T00:38:30.0338130Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T00:38:30.0373560Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T00:38:30.0564357Z Entering 'third_party/pocketfft' 2025-10-10T00:38:30.0596309Z Entering 'third_party/protobuf' 2025-10-10T00:38:30.0664780Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T00:38:30.0696381Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T00:38:30.0739665Z Entering 'third_party/psimd' 2025-10-10T00:38:30.0773296Z Entering 'third_party/pthreadpool' 2025-10-10T00:38:30.0800841Z Entering 'third_party/pybind11' 2025-10-10T00:38:30.0832068Z Entering 'third_party/python-peachpy' 2025-10-10T00:38:30.0874241Z Entering 'third_party/sleef' 2025-10-10T00:38:30.0907699Z Entering 'third_party/tensorpipe' 2025-10-10T00:38:30.0942564Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T00:38:30.0971815Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T00:38:30.1001287Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T00:38:30.1036745Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T00:38:30.1073971Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T00:38:30.1196116Z Prepare all required actions 2025-10-10T00:38:30.1196584Z Getting action download info 2025-10-10T00:38:30.2564898Z ##[group]Run ./.github/actions/setup-linux 2025-10-10T00:38:30.2565146Z env: 2025-10-10T00:38:30.2565306Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.2565479Z ##[endgroup] 2025-10-10T00:38:30.2604183Z ##[group]Run set -euo pipefail 2025-10-10T00:38:30.2604463Z set -euo pipefail 2025-10-10T00:38:30.2604672Z function get_ec2_metadata() { 2025-10-10T00:38:30.2604923Z  # Pulled from instance metadata endpoint for EC2 2025-10-10T00:38:30.2605335Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-10-10T00:38:30.2605679Z  category=$1 2025-10-10T00:38:30.2605921Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-10-10T00:38:30.2606191Z  runner_name_str=i-01dad1d3a0e80f7c0 2025-10-10T00:38:30.2606468Z  if [[ -f /.inarc ]]; then 2025-10-10T00:38:30.2606696Z  echo "ARC Runner, no info on ec2 metadata" 2025-10-10T00:38:30.2606937Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-10-10T00:38:30.2607242Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-10-10T00:38:30.2607510Z  else 2025-10-10T00:38:30.2608038Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-10-10T00:38:30.2608565Z  fi 2025-10-10T00:38:30.2608718Z } 2025-10-10T00:38:30.2608912Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-10-10T00:38:30.2609193Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-10-10T00:38:30.2609498Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-10-10T00:38:30.2609756Z echo "system info $(uname -a)" 2025-10-10T00:38:30.2615662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:30.2615913Z env: 2025-10-10T00:38:30.2616082Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.2616273Z ##[endgroup] 2025-10-10T00:38:30.2756819Z ami-id: ami-08982f1c5bf93d976 2025-10-10T00:38:30.2861315Z instance-id: i-01dad1d3a0e80f7c0 2025-10-10T00:38:30.3000613Z instance-type: m7i-flex.8xlarge 2025-10-10T00:38:30.3011985Z system info Linux ip-10-0-32-50.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-10-10T00:38:30.3043642Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-10-10T00:38:30.3043992Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-10-10T00:38:30.3049505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:30.3049748Z env: 2025-10-10T00:38:30.3049911Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.3050097Z ##[endgroup] 2025-10-10T00:38:30.3099372Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:38:30.3099980Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:38:30.3104462Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:30.3104734Z env: 2025-10-10T00:38:30.3104913Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.3105118Z ##[endgroup] 2025-10-10T00:38:30.3161374Z ##[group]Run if systemctl is-active --quiet docker; then 2025-10-10T00:38:30.3161678Z if systemctl is-active --quiet docker; then 2025-10-10T00:38:30.3161938Z  echo "Docker daemon is running..."; 2025-10-10T00:38:30.3162157Z else 2025-10-10T00:38:30.3162394Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-10-10T00:38:30.3162667Z fi 2025-10-10T00:38:30.3166629Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:30.3166871Z env: 2025-10-10T00:38:30.3167025Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.3167210Z ##[endgroup] 2025-10-10T00:38:30.3244602Z Docker daemon is running... 2025-10-10T00:38:30.3281544Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T00:38:30.3281775Z with: 2025-10-10T00:38:30.3281933Z shell: bash 2025-10-10T00:38:30.3282109Z timeout_minutes: 5 2025-10-10T00:38:30.3282292Z max_attempts: 3 2025-10-10T00:38:30.3282479Z retry_wait_seconds: 30 2025-10-10T00:38:30.3283921Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-10-10T00:38:30.3285294Z polling_interval_seconds: 1 2025-10-10T00:38:30.3285508Z warning_on_retry: true 2025-10-10T00:38:30.3285698Z continue_on_error: false 2025-10-10T00:38:30.3285893Z env: 2025-10-10T00:38:30.3286062Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:30.3286264Z AWS_RETRY_MODE: standard 2025-10-10T00:38:30.3286458Z AWS_MAX_ATTEMPTS: 5 2025-10-10T00:38:30.3286653Z AWS_DEFAULT_REGION: us-east-1 2025-10-10T00:38:30.3286856Z ##[endgroup] 2025-10-10T00:38:31.3274033Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T00:38:31.3277003Z Configure a credential helper to remove this warning. See 2025-10-10T00:38:31.3277482Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T00:38:31.3277755Z 2025-10-10T00:38:31.3277833Z Login Succeeded 2025-10-10T00:38:31.4688386Z Command completed after 1 attempt(s). 2025-10-10T00:38:31.4776901Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:38:31.4777263Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:38:31.4777564Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:38:31.4785449Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:31.4785709Z env: 2025-10-10T00:38:31.4785887Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:31.4786082Z ##[endgroup] 2025-10-10T00:38:31.4882495Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T00:38:31.4882884Z # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T00:38:31.4883163Z # shellcheck disable=SC2046 2025-10-10T00:38:31.4883406Z docker stop $(docker ps -q) || true 2025-10-10T00:38:31.4883626Z # Prune all of the docker images 2025-10-10T00:38:31.4883854Z docker system prune -af 2025-10-10T00:38:31.4888552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:31.4888804Z env: 2025-10-10T00:38:31.4888960Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:31.4889272Z ##[endgroup] 2025-10-10T00:38:31.5381655Z "docker stop" requires at least 1 argument. 2025-10-10T00:38:31.5381993Z See 'docker stop --help'. 2025-10-10T00:38:31.5382128Z 2025-10-10T00:38:31.5382259Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-10-10T00:38:31.5382449Z 2025-10-10T00:38:31.5382536Z Stop one or more running containers 2025-10-10T00:38:31.5549807Z Total reclaimed space: 0B 2025-10-10T00:38:31.5691400Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-10-10T00:38:31.5691715Z with: 2025-10-10T00:38:31.5692294Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5692941Z use-custom-docker-registry: true 2025-10-10T00:38:31.5693182Z docker-build-dir: .ci/docker 2025-10-10T00:38:31.5693404Z docker-build-script: ./build.sh 2025-10-10T00:38:31.5693626Z working-directory: . 2025-10-10T00:38:31.5693905Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5694192Z force-push: false 2025-10-10T00:38:31.5694367Z env: 2025-10-10T00:38:31.5694542Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:31.5694742Z ##[endgroup] 2025-10-10T00:38:31.5716402Z ##[group]Run set -ex 2025-10-10T00:38:31.5716649Z set -ex 2025-10-10T00:38:31.5716828Z  2025-10-10T00:38:31.5717171Z # If the docker build directory or the build script doesn't exist, the action will 2025-10-10T00:38:31.5717613Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-10-10T00:38:31.5717990Z # job could then download the pre-built image as usual 2025-10-10T00:38:31.5718448Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-10-10T00:38:31.5718872Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5719121Z else 2025-10-10T00:38:31.5719325Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5719634Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5719913Z  2025-10-10T00:38:31.5720294Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-10-10T00:38:31.5720722Z  exit 0 2025-10-10T00:38:31.5720891Z fi 2025-10-10T00:38:31.5721061Z  2025-10-10T00:38:31.5721318Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-10-10T00:38:31.5721738Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-10-10T00:38:31.5722099Z  # use it as it is, but first let's extract the tag 2025-10-10T00:38:31.5722424Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-10-10T00:38:31.5722774Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5723106Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5723381Z else 2025-10-10T00:38:31.5723587Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-10-10T00:38:31.5723857Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-10-10T00:38:31.5724148Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-10-10T00:38:31.5724393Z  fi 2025-10-10T00:38:31.5724720Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-10-10T00:38:31.5725131Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5725572Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5726016Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5726410Z fi 2025-10-10T00:38:31.5732171Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:31.5732439Z env: 2025-10-10T00:38:31.5732627Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:31.5732841Z REPO_NAME: pytorch 2025-10-10T00:38:31.5733567Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5734175Z DOCKER_BUILD_DIR: .ci/docker 2025-10-10T00:38:31.5734393Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-10-10T00:38:31.5734678Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5734970Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-10-10T00:38:31.5735190Z CUSTOM_TAG_PREFIX: 2025-10-10T00:38:31.5735377Z ##[endgroup] 2025-10-10T00:38:31.5759678Z + [[ -d .ci/docker ]] 2025-10-10T00:38:31.5765579Z + [[ -f .ci/docker/./build.sh ]] 2025-10-10T00:38:31.5771218Z + [[ true == \t\r\u\e ]] 2025-10-10T00:38:31.5773241Z + echo skip=false 2025-10-10T00:38:31.5774414Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-10-10T00:38:31.5775676Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5777778Z ++ awk -F '[:,]' '{print $2}' 2025-10-10T00:38:31.5790574Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5793381Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5794227Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5821412Z ##[group]Run set +e 2025-10-10T00:38:31.5821657Z set +e 2025-10-10T00:38:31.5821835Z set -x 2025-10-10T00:38:31.5821994Z  2025-10-10T00:38:31.5822159Z login() { 2025-10-10T00:38:31.5822493Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-10-10T00:38:31.5822847Z } 2025-10-10T00:38:31.5823002Z  2025-10-10T00:38:31.5823161Z retry () { 2025-10-10T00:38:31.5823363Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-10-10T00:38:31.5823585Z } 2025-10-10T00:38:31.5823733Z  2025-10-10T00:38:31.5823905Z retry login "${DOCKER_REGISTRY}" 2025-10-10T00:38:31.5824120Z  2025-10-10T00:38:31.5824285Z START_TIME=$(date +%s) 2025-10-10T00:38:31.5824495Z # Wait up to 120 minutes 2025-10-10T00:38:31.5824759Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-10-10T00:38:31.5825102Z  # Check if image already exists, if it does then skip building it 2025-10-10T00:38:31.5825440Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-10-10T00:38:31.5825690Z  exit 0 2025-10-10T00:38:31.5825868Z  fi 2025-10-10T00:38:31.5826035Z  2025-10-10T00:38:31.5826618Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-10-10T00:38:31.5827059Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-10-10T00:38:31.5827487Z  # latter, it will wait for the Docker images to become available before continuing 2025-10-10T00:38:31.5827848Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-10-10T00:38:31.5828139Z  # It's a Docker build job, let's build the image 2025-10-10T00:38:31.5828385Z  break 2025-10-10T00:38:31.5828566Z  else 2025-10-10T00:38:31.5828925Z  # It's a regular build job, wait for the image to become available 2025-10-10T00:38:31.5829209Z  sleep 300 2025-10-10T00:38:31.5829387Z  fi 2025-10-10T00:38:31.5829538Z done 2025-10-10T00:38:31.5829696Z  2025-10-10T00:38:31.5829936Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-10-10T00:38:31.5830356Z # be empty. The default action would be to continue rebuild the image 2025-10-10T00:38:31.5830695Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-10-10T00:38:31.5830992Z  # if we're on the base branch then use the parent commit 2025-10-10T00:38:31.5831298Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-10-10T00:38:31.5831801Z else 2025-10-10T00:38:31.5832054Z  # otherwise we're on a PR, so use the most recent base commit 2025-10-10T00:38:31.5832398Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-10-10T00:38:31.5832651Z fi 2025-10-10T00:38:31.5832810Z  2025-10-10T00:38:31.5832995Z if [[ -z "${MERGE_BASE}" ]]; then 2025-10-10T00:38:31.5833256Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5833495Z  2025-10-10T00:38:31.5833814Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-10-10T00:38:31.5834181Z  exit 0 2025-10-10T00:38:31.5834359Z fi 2025-10-10T00:38:31.5834523Z  2025-10-10T00:38:31.5834744Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-10-10T00:38:31.5835196Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-10-10T00:38:31.5835577Z  exit 1 2025-10-10T00:38:31.5835752Z fi 2025-10-10T00:38:31.5835912Z  2025-10-10T00:38:31.5836166Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-10-10T00:38:31.5836603Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-10-10T00:38:31.5836991Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-10-10T00:38:31.5837433Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-10-10T00:38:31.5837938Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-10-10T00:38:31.5838233Z fi 2025-10-10T00:38:31.5871604Z  2025-10-10T00:38:31.5871841Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-10-10T00:38:31.5877650Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:31.5878005Z env: 2025-10-10T00:38:31.5878185Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:31.5878404Z DOCKER_BUILD_DIR: .ci/docker 2025-10-10T00:38:31.5878667Z BASE_REVISION: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:38:31.5879336Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5880144Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:31.5880621Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5880904Z DOCKER_PUSH: 2025-10-10T00:38:31.5881088Z ##[endgroup] 2025-10-10T00:38:31.5913824Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5914183Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5915399Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:31.5915771Z + aws ecr get-login-password --region us-east-1 2025-10-10T00:38:32.0171062Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T00:38:32.0171710Z Login Succeeded 2025-10-10T00:38:32.0172101Z Configure a credential helper to remove this warning. See 2025-10-10T00:38:32.0172568Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T00:38:32.0175418Z 2025-10-10T00:38:32.0193101Z ++ date +%s 2025-10-10T00:38:32.0203480Z + START_TIME=1760056712 2025-10-10T00:38:32.0204422Z ++ date +%s 2025-10-10T00:38:32.0216145Z + [[ 1760049512 -lt 1760056712 ]] 2025-10-10T00:38:32.0216843Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:32.2593522Z { 2025-10-10T00:38:32.2593754Z "schemaVersion": 2, 2025-10-10T00:38:32.2597493Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-10-10T00:38:32.2597802Z "config": { 2025-10-10T00:38:32.2598034Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-10-10T00:38:32.2598306Z "size": 30298, 2025-10-10T00:38:32.2598634Z "digest": "sha256:be8feb4482de208ab33ff6be61032fcdbeafec87bebc633426f293b292801f69" 2025-10-10T00:38:32.2598934Z }, 2025-10-10T00:38:32.2599070Z "layers": [ 2025-10-10T00:38:32.2599220Z { 2025-10-10T00:38:32.2599441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2599701Z "size": 30447990, 2025-10-10T00:38:32.2599981Z "digest": "sha256:828c1365039a657352c737a62d13e1932951b5658eb6bd9b9096ea9b73562453" 2025-10-10T00:38:32.2600270Z }, 2025-10-10T00:38:32.2600412Z { 2025-10-10T00:38:32.2600634Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2600890Z "size": 1554, 2025-10-10T00:38:32.2601164Z "digest": "sha256:d48484dff770069589ea855e7f430bbe978d639e71d7a54efb1e193d710984b7" 2025-10-10T00:38:32.2601451Z }, 2025-10-10T00:38:32.2604707Z { 2025-10-10T00:38:32.2604952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2605216Z "size": 313650057, 2025-10-10T00:38:32.2605526Z "digest": "sha256:63f2ff1b73caf24b8c6b89e94c648d9f2903c07db1e3f03f8c53b2f4cdb88b23" 2025-10-10T00:38:32.2605831Z }, 2025-10-10T00:38:32.2605966Z { 2025-10-10T00:38:32.2606172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2606424Z "size": 792, 2025-10-10T00:38:32.2606709Z "digest": "sha256:4a952c1cd3ddc728ce55cef2c0f8a2cead9edf99d11f3b5a0cf843bf8b58aab3" 2025-10-10T00:38:32.2607006Z }, 2025-10-10T00:38:32.2607135Z { 2025-10-10T00:38:32.2607347Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2607607Z "size": 106, 2025-10-10T00:38:32.2607879Z "digest": "sha256:cf3649f6486ae5ce318a0e0c72a096aee44e4fb743f2d4adb473fc308b7e96ac" 2025-10-10T00:38:32.2608169Z }, 2025-10-10T00:38:32.2608307Z { 2025-10-10T00:38:32.2608519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2608777Z "size": 703, 2025-10-10T00:38:32.2609037Z "digest": "sha256:7a4d1eb936e8e4e9eca13c9b423660a1d98080dd970e416961d5899ab2bada4e" 2025-10-10T00:38:32.2609325Z }, 2025-10-10T00:38:32.2609461Z { 2025-10-10T00:38:32.2609672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2609923Z "size": 1212, 2025-10-10T00:38:32.2610194Z "digest": "sha256:fec720215749af5500d26baeb404c58afc3bbfa548de6d4fdd729c288e25187d" 2025-10-10T00:38:32.2610497Z }, 2025-10-10T00:38:32.2610634Z { 2025-10-10T00:38:32.2610839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2611095Z "size": 483, 2025-10-10T00:38:32.2611364Z "digest": "sha256:130af8bfbe27c7f09b41f241968e2df384861e6af82e8ff4a0a7acf82652b7ea" 2025-10-10T00:38:32.2611657Z }, 2025-10-10T00:38:32.2611863Z { 2025-10-10T00:38:32.2612098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2612436Z + exit 0 2025-10-10T00:38:32.2612591Z "size": 110343733, 2025-10-10T00:38:32.2613155Z "digest": "sha256:193b2b1a9925e076c5c5a070b1a2a5656f771dbbe6fe54e24858e214c999ec31" 2025-10-10T00:38:32.2613436Z }, 2025-10-10T00:38:32.2613573Z { 2025-10-10T00:38:32.2613788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2614049Z "size": 5033, 2025-10-10T00:38:32.2614318Z "digest": "sha256:38d02879e15dfacf9075db83626b0a9b06e7b100dfc8b19e2d8aac0ed82a1d7b" 2025-10-10T00:38:32.2614747Z }, 2025-10-10T00:38:32.2614924Z { 2025-10-10T00:38:32.2615173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2615434Z "size": 1709, 2025-10-10T00:38:32.2615697Z "digest": "sha256:fb4547453b1c476b51e9d234ffb01bbafea460abcd5ec7e3d6f547339285312d" 2025-10-10T00:38:32.2615999Z }, 2025-10-10T00:38:32.2616139Z { 2025-10-10T00:38:32.2616355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2616605Z "size": 724, 2025-10-10T00:38:32.2616869Z "digest": "sha256:5309e360108e00b09730355a8aa7fed622a6f8bca2b76d0ff01498a5a6fabab6" 2025-10-10T00:38:32.2617158Z }, 2025-10-10T00:38:32.2617292Z { 2025-10-10T00:38:32.2617500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2617761Z "size": 542, 2025-10-10T00:38:32.2618033Z "digest": "sha256:9be330fbd7beead7b21b290a066dac5164e7d36ad14130818f41b5eba2a188b6" 2025-10-10T00:38:32.2618329Z }, 2025-10-10T00:38:32.2618549Z { 2025-10-10T00:38:32.2618820Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2619089Z "size": 3422780532, 2025-10-10T00:38:32.2619376Z "digest": "sha256:2cf6366f7d1178da512fd5c896f31b70b31cbf6144b5e5d6ee3737bcc79c5bcc" 2025-10-10T00:38:32.2619666Z }, 2025-10-10T00:38:32.2619807Z { 2025-10-10T00:38:32.2620027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2620290Z "size": 32, 2025-10-10T00:38:32.2620560Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2620880Z }, 2025-10-10T00:38:32.2621021Z { 2025-10-10T00:38:32.2621251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2621518Z "size": 380, 2025-10-10T00:38:32.2621793Z "digest": "sha256:56c4f51a013172b027f025ba48a44821c49a7c62e3144d38779828aa1a830cba" 2025-10-10T00:38:32.2622103Z }, 2025-10-10T00:38:32.2622247Z { 2025-10-10T00:38:32.2622469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2622746Z "size": 236627, 2025-10-10T00:38:32.2623049Z "digest": "sha256:f7b7f96144123201eded917df066aa1045bc5753eac2eee567d84001c2e94f9b" 2025-10-10T00:38:32.2623357Z }, 2025-10-10T00:38:32.2623494Z { 2025-10-10T00:38:32.2623722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2624006Z "size": 232, 2025-10-10T00:38:32.2624282Z "digest": "sha256:70aeb425e04c172dec92d29b369f6e55c7b017991bd7396b7685e743da0b8553" 2025-10-10T00:38:32.2624588Z }, 2025-10-10T00:38:32.2624734Z { 2025-10-10T00:38:32.2624958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2625244Z "size": 3539945, 2025-10-10T00:38:32.2625526Z "digest": "sha256:97346936eb304dc3eb592147cb957c91e4b9ea86d11bab69cc8cff224f61491b" 2025-10-10T00:38:32.2625839Z }, 2025-10-10T00:38:32.2625981Z { 2025-10-10T00:38:32.2626278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2626561Z "size": 1478, 2025-10-10T00:38:32.2626850Z "digest": "sha256:9e5860b535d4fa06b166de3400e69fa4b20d8cc6c23f78fdc7934c0a6d1f1714" 2025-10-10T00:38:32.2627170Z }, 2025-10-10T00:38:32.2627311Z { 2025-10-10T00:38:32.2627527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2627853Z "size": 483, 2025-10-10T00:38:32.2628137Z "digest": "sha256:0ba1a45e12052d52ea8f656694d498f71f0a69eff39bbd947bbce968a572d8a3" 2025-10-10T00:38:32.2628445Z }, 2025-10-10T00:38:32.2628584Z { 2025-10-10T00:38:32.2628870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2629151Z "size": 200, 2025-10-10T00:38:32.2629443Z "digest": "sha256:3b7d78f74bbea51b906e4ea15594fe2acb7d9e3043958b83e055a634094f95a4" 2025-10-10T00:38:32.2629754Z }, 2025-10-10T00:38:32.2629903Z { 2025-10-10T00:38:32.2630136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2630468Z "size": 605, 2025-10-10T00:38:32.2630755Z "digest": "sha256:6beb9b78ab6eaddf4fa53a570a1340dcc12f366d89c43be8132808756138f673" 2025-10-10T00:38:32.2631077Z }, 2025-10-10T00:38:32.2631227Z { 2025-10-10T00:38:32.2631589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2631878Z "size": 226, 2025-10-10T00:38:32.2632172Z "digest": "sha256:25a0a5e5e819c24a72a56f7db69ddb6304abfb26540096fb7effe392be837bbf" 2025-10-10T00:38:32.2632494Z }, 2025-10-10T00:38:32.2632648Z { 2025-10-10T00:38:32.2632937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2633227Z "size": 828, 2025-10-10T00:38:32.2633513Z "digest": "sha256:5227fe91454bb5a30ab04a28c7528d30834ee393e03a1ee7bb0747c3b2814145" 2025-10-10T00:38:32.2633829Z }, 2025-10-10T00:38:32.2633970Z { 2025-10-10T00:38:32.2634207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2634489Z "size": 32, 2025-10-10T00:38:32.2634785Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2635100Z }, 2025-10-10T00:38:32.2635274Z { 2025-10-10T00:38:32.2635504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2635822Z "size": 104, 2025-10-10T00:38:32.2636113Z "digest": "sha256:66f9a01572e213b55fcf8dfee3a1053033513464bcbcc99bdf04e1937480e85a" 2025-10-10T00:38:32.2636423Z }, 2025-10-10T00:38:32.2636553Z { 2025-10-10T00:38:32.2636746Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2636999Z "size": 1495, 2025-10-10T00:38:32.2637257Z "digest": "sha256:c252c4979145558e389f3edb86fb9dd186143778e8b7382cfb95e45ea94cd8c9" 2025-10-10T00:38:32.2637528Z }, 2025-10-10T00:38:32.2637649Z { 2025-10-10T00:38:32.2637852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2638098Z "size": 453793745, 2025-10-10T00:38:32.2638367Z "digest": "sha256:f9c84d68915b83e3bd4b5bcde6de24b70f80e1f802b7db3987034127cdedd62a" 2025-10-10T00:38:32.2638633Z }, 2025-10-10T00:38:32.2638761Z { 2025-10-10T00:38:32.2638961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2639203Z "size": 163, 2025-10-10T00:38:32.2639455Z "digest": "sha256:a086fb2969ce3f7dddd9784a2eafad77e02f7fb9a02fcc68f2a37063eebd9f24" 2025-10-10T00:38:32.2639735Z }, 2025-10-10T00:38:32.2639863Z { 2025-10-10T00:38:32.2640064Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2640300Z "size": 346, 2025-10-10T00:38:32.2640554Z "digest": "sha256:a6c00b8f11a4afce98fb31c01839812e4c2283b827d77854a2e613b4e26ab1fb" 2025-10-10T00:38:32.2640831Z }, 2025-10-10T00:38:32.2640960Z { 2025-10-10T00:38:32.2641153Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2641396Z "size": 32, 2025-10-10T00:38:32.2641655Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2641931Z }, 2025-10-10T00:38:32.2642053Z { 2025-10-10T00:38:32.2642256Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2642501Z "size": 106, 2025-10-10T00:38:32.2642744Z "digest": "sha256:fc80c2144458a32870857c2f23414d3566f213e41554b0f769633e6a05a1abf0" 2025-10-10T00:38:32.2643006Z }, 2025-10-10T00:38:32.2643135Z { 2025-10-10T00:38:32.2643338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2643584Z "size": 425, 2025-10-10T00:38:32.2643827Z "digest": "sha256:3894d677ce34d9b9eb91c1191c95f932a092ea577c9f1bbec99a48d3f5c35e89" 2025-10-10T00:38:32.2644184Z }, 2025-10-10T00:38:32.2644316Z { 2025-10-10T00:38:32.2644518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2644763Z "size": 19309405, 2025-10-10T00:38:32.2645023Z "digest": "sha256:be6227b9675ebe173490450a55960904a03ce48a88b05eee93f990d1000075a9" 2025-10-10T00:38:32.2645355Z }, 2025-10-10T00:38:32.2645489Z { 2025-10-10T00:38:32.2645682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2645926Z "size": 108, 2025-10-10T00:38:32.2646174Z "digest": "sha256:c5731ed75d330627291fb79a6dcdaf52400f1356d00d3dbb77bb6dcca61fbec9" 2025-10-10T00:38:32.2646447Z }, 2025-10-10T00:38:32.2646567Z { 2025-10-10T00:38:32.2646768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2647010Z "size": 638, 2025-10-10T00:38:32.2647258Z "digest": "sha256:e3d52ff40b4c33e27b1be81f97636b9a3c52fd51829083408f427e3133ae82ba" 2025-10-10T00:38:32.2647526Z }, 2025-10-10T00:38:32.2647654Z { 2025-10-10T00:38:32.2647856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2648098Z "size": 724, 2025-10-10T00:38:32.2648337Z "digest": "sha256:5309e360108e00b09730355a8aa7fed622a6f8bca2b76d0ff01498a5a6fabab6" 2025-10-10T00:38:32.2648611Z }, 2025-10-10T00:38:32.2648739Z { 2025-10-10T00:38:32.2648945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2649182Z "size": 148, 2025-10-10T00:38:32.2649429Z "digest": "sha256:86ca23246f244a84a7ef07bff072512ba3c292399f2d4606e6ae9e0f6c4962a6" 2025-10-10T00:38:32.2649699Z }, 2025-10-10T00:38:32.2649830Z { 2025-10-10T00:38:32.2650021Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2650264Z "size": 134, 2025-10-10T00:38:32.2650508Z "digest": "sha256:40e2321b482aa888683c5618b7e59f160fbee717ba296a30d5ae6d36856f7180" 2025-10-10T00:38:32.2650779Z }, 2025-10-10T00:38:32.2650899Z { 2025-10-10T00:38:32.2651097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2651339Z "size": 140, 2025-10-10T00:38:32.2651590Z "digest": "sha256:b25589a402efcc475c0dde161918cfe2626afb76015a61eeab4f3eed0e560a86" 2025-10-10T00:38:32.2651859Z }, 2025-10-10T00:38:32.2651989Z { 2025-10-10T00:38:32.2652193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2652442Z "size": 18656543993, 2025-10-10T00:38:32.2652699Z "digest": "sha256:3be3b4544852544b087a07c4d0a514f9821aa2bd198aecbd64b1e302acaaa280" 2025-10-10T00:38:32.2652970Z }, 2025-10-10T00:38:32.2653097Z { 2025-10-10T00:38:32.2653299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2653535Z "size": 223, 2025-10-10T00:38:32.2653791Z "digest": "sha256:ca8d18c1fad63dd7f3c9b3fbcf66e20aa1d6efc508d489aa2c9ec7b866f293b5" 2025-10-10T00:38:32.2654074Z }, 2025-10-10T00:38:32.2654208Z { 2025-10-10T00:38:32.2654403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2654682Z "size": 312300631, 2025-10-10T00:38:32.2654959Z "digest": "sha256:ad03627b7e8e3efca5bb0986ceee5a1eec9cbef1c626508bb6eca77743900d8b" 2025-10-10T00:38:32.2655234Z }, 2025-10-10T00:38:32.2655353Z { 2025-10-10T00:38:32.2655551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2655792Z "size": 6535964311, 2025-10-10T00:38:32.2656054Z "digest": "sha256:b7b658babd8c3040dfd62cf72ec1e7a85f4c10ac8951694314b61ceebca39201" 2025-10-10T00:38:32.2656317Z }, 2025-10-10T00:38:32.2656446Z { 2025-10-10T00:38:32.2656640Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2656877Z "size": 129, 2025-10-10T00:38:32.2657119Z "digest": "sha256:784eefe614e4a04f748fbb06028322fed5e1dd2efe3fccbe58020b73f8fe86b5" 2025-10-10T00:38:32.2657393Z }, 2025-10-10T00:38:32.2657522Z { 2025-10-10T00:38:32.2657809Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2658037Z "size": 881, 2025-10-10T00:38:32.2658280Z "digest": "sha256:9084ec9ef863cd90fc11f1f779e5f51181e24992065774707ff19702b8ca6112" 2025-10-10T00:38:32.2658542Z }, 2025-10-10T00:38:32.2658670Z { 2025-10-10T00:38:32.2658863Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2659155Z "size": 724, 2025-10-10T00:38:32.2659403Z "digest": "sha256:5309e360108e00b09730355a8aa7fed622a6f8bca2b76d0ff01498a5a6fabab6" 2025-10-10T00:38:32.2659675Z }, 2025-10-10T00:38:32.2659795Z { 2025-10-10T00:38:32.2659997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2660240Z "size": 139, 2025-10-10T00:38:32.2660481Z "digest": "sha256:0a890a7dda707f271122efd5be84d7097a418647980d059657837fa684a2e28f" 2025-10-10T00:38:32.2660745Z }, 2025-10-10T00:38:32.2660873Z { 2025-10-10T00:38:32.2661075Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2661319Z "size": 32, 2025-10-10T00:38:32.2661564Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2661839Z }, 2025-10-10T00:38:32.2661972Z { 2025-10-10T00:38:32.2662175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2662419Z "size": 159, 2025-10-10T00:38:32.2662673Z "digest": "sha256:5907e244ca8505c5638d1ea7b656058137d1ff766c5ce1ae8fb290147a06801e" 2025-10-10T00:38:32.2662949Z }, 2025-10-10T00:38:32.2663081Z { 2025-10-10T00:38:32.2663280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2663531Z "size": 1010, 2025-10-10T00:38:32.2663796Z "digest": "sha256:7d9cc7db846bfdd412dff1edf9db939924a857b6557ffc59c331215b6092372e" 2025-10-10T00:38:32.2664082Z }, 2025-10-10T00:38:32.2664211Z { 2025-10-10T00:38:32.2664420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2664684Z "size": 724, 2025-10-10T00:38:32.2664942Z "digest": "sha256:5309e360108e00b09730355a8aa7fed622a6f8bca2b76d0ff01498a5a6fabab6" 2025-10-10T00:38:32.2665228Z }, 2025-10-10T00:38:32.2665357Z { 2025-10-10T00:38:32.2665569Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2665826Z "size": 135, 2025-10-10T00:38:32.2666093Z "digest": "sha256:c6f4775b798588c57dd98c6eb3b1d32aa4468c19b37a15be2bf3eec7a603f515" 2025-10-10T00:38:32.2666584Z }, 2025-10-10T00:38:32.2666741Z { 2025-10-10T00:38:32.2666974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2667251Z "size": 32, 2025-10-10T00:38:32.2667517Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2667795Z }, 2025-10-10T00:38:32.2667923Z { 2025-10-10T00:38:32.2668113Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2668363Z "size": 158, 2025-10-10T00:38:32.2668625Z "digest": "sha256:ed36b9de7242b4878ac79a2c88cf895eedf22b15f22ca3f1d6a515487aef3a11" 2025-10-10T00:38:32.2668904Z }, 2025-10-10T00:38:32.2669037Z { 2025-10-10T00:38:32.2669232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2669479Z "size": 1369, 2025-10-10T00:38:32.2669734Z "digest": "sha256:cfc72d1580d6d4c6df8da5089b4374623f5d126a06f2ed878a20ec1c8e9352ff" 2025-10-10T00:38:32.2670019Z }, 2025-10-10T00:38:32.2670137Z { 2025-10-10T00:38:32.2670333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2670571Z "size": 32, 2025-10-10T00:38:32.2670813Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2671074Z }, 2025-10-10T00:38:32.2671198Z { 2025-10-10T00:38:32.2671395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2671631Z "size": 137, 2025-10-10T00:38:32.2671860Z "digest": "sha256:140888b3776d088557dc916199b13ec55678b4f21de40dc757be4a0e6ff30d44" 2025-10-10T00:38:32.2672169Z }, 2025-10-10T00:38:32.2672298Z { 2025-10-10T00:38:32.2672499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2672735Z "size": 527, 2025-10-10T00:38:32.2672991Z "digest": "sha256:43ee72be7eaae6b5387aebcb855e9700c389eaf5af12731a0f023f6fd55aa3a4" 2025-10-10T00:38:32.2673276Z }, 2025-10-10T00:38:32.2673491Z { 2025-10-10T00:38:32.2673693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2673936Z "size": 32, 2025-10-10T00:38:32.2674191Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2674468Z }, 2025-10-10T00:38:32.2674594Z { 2025-10-10T00:38:32.2674796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2675044Z "size": 104, 2025-10-10T00:38:32.2675301Z "digest": "sha256:4b724c5636ce744f36ecdd4e199f7fe417bba46126f15c312eb4cc63cf5d53b7" 2025-10-10T00:38:32.2675578Z }, 2025-10-10T00:38:32.2675709Z { 2025-10-10T00:38:32.2675914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2676160Z "size": 430, 2025-10-10T00:38:32.2676401Z "digest": "sha256:47122b9b853f72029a5ef9cd276bcc526d8f3a40c94319157ae7f36782401a54" 2025-10-10T00:38:32.2676674Z }, 2025-10-10T00:38:32.2676817Z { 2025-10-10T00:38:32.2677020Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2677256Z "size": 32, 2025-10-10T00:38:32.2677510Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2677786Z }, 2025-10-10T00:38:32.2677916Z { 2025-10-10T00:38:32.2678109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2678352Z "size": 109, 2025-10-10T00:38:32.2678606Z "digest": "sha256:501ccd6f6912be09145cb83a4221553fe5c2cc8a98b71062d2c5c3a5b8fd6435" 2025-10-10T00:38:32.2678880Z }, 2025-10-10T00:38:32.2679005Z { 2025-10-10T00:38:32.2679206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2679453Z "size": 1896, 2025-10-10T00:38:32.2679703Z "digest": "sha256:7c29f290a530d549084ffe614356549f235c3e0f58929a6ca6618a721bf70e6a" 2025-10-10T00:38:32.2679967Z }, 2025-10-10T00:38:32.2680095Z { 2025-10-10T00:38:32.2680299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2680543Z "size": 244502157, 2025-10-10T00:38:32.2680800Z "digest": "sha256:6f129b7d3f62373779b4c9fd4a451a2bd468cdbf3e7a9635371e99b3caa4dd3b" 2025-10-10T00:38:32.2681075Z }, 2025-10-10T00:38:32.2681202Z { 2025-10-10T00:38:32.2681403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2681639Z "size": 106, 2025-10-10T00:38:32.2681888Z "digest": "sha256:f4f4042795eab8ed78bfc3855ceb875058c12d9131335b1982663081cad26c0d" 2025-10-10T00:38:32.2682157Z }, 2025-10-10T00:38:32.2682286Z { 2025-10-10T00:38:32.2682482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2682726Z "size": 164, 2025-10-10T00:38:32.2682980Z "digest": "sha256:a3c70cd222f0ffcd98ea6b7f94b812f1e2df67c35ea34522b3a94d32969d9c37" 2025-10-10T00:38:32.2683259Z }, 2025-10-10T00:38:32.2683380Z { 2025-10-10T00:38:32.2683583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2683832Z "size": 7943, 2025-10-10T00:38:32.2684091Z "digest": "sha256:0d733ca581b2fc05d6cd6ab09d10ba0d179ceb13d01b98861e5823caa1d4fe1c" 2025-10-10T00:38:32.2684362Z }, 2025-10-10T00:38:32.2684494Z { 2025-10-10T00:38:32.2684695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2684942Z "size": 8072, 2025-10-10T00:38:32.2685190Z "digest": "sha256:23e94f3ab9a7d00135247c0fbf79f69b8b201fc5b30e69fa3fe561831044cc36" 2025-10-10T00:38:32.2685470Z }, 2025-10-10T00:38:32.2685599Z { 2025-10-10T00:38:32.2685800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2686088Z "size": 304, 2025-10-10T00:38:32.2686351Z "digest": "sha256:ee028affbc09ac8ee9cc437c341bb81c09df9e6bfd1bdd5da87809d24ebf3d05" 2025-10-10T00:38:32.2686637Z }, 2025-10-10T00:38:32.2686764Z { 2025-10-10T00:38:32.2686961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2687207Z "size": 32, 2025-10-10T00:38:32.2687502Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2687783Z }, 2025-10-10T00:38:32.2687906Z { 2025-10-10T00:38:32.2688108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2688357Z "size": 108, 2025-10-10T00:38:32.2688605Z "digest": "sha256:f090f773ba0e46e96a12583445cf1f6741e20aea9362c49517163d0f0bfc0fc1" 2025-10-10T00:38:32.2688872Z }, 2025-10-10T00:38:32.2689000Z { 2025-10-10T00:38:32.2689205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2689460Z "size": 54145665, 2025-10-10T00:38:32.2689711Z "digest": "sha256:56f1f8e7f2476d5d6777b43002615dd64f62ba9a6a82678195124a491048faa0" 2025-10-10T00:38:32.2689982Z }, 2025-10-10T00:38:32.2690111Z { 2025-10-10T00:38:32.2690313Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-10-10T00:38:32.2690551Z "size": 32, 2025-10-10T00:38:32.2690807Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-10-10T00:38:32.2691085Z } 2025-10-10T00:38:32.2691216Z ] 2025-10-10T00:38:32.2691341Z } 2025-10-10T00:38:32.2719875Z ##[group]Run set -eux 2025-10-10T00:38:32.2720112Z set -eux 2025-10-10T00:38:32.2720413Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-10-10T00:38:32.2721183Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-10-10T00:38:32.2727627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:32.2727885Z env: 2025-10-10T00:38:32.2728065Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:32.2728262Z ##[endgroup] 2025-10-10T00:38:32.2758960Z + jq --raw-output .SecretString 2025-10-10T00:38:32.2759365Z + docker login --username pytorchbot --password-stdin 2025-10-10T00:38:32.2766081Z + jq -r .docker_hub_readonly_token 2025-10-10T00:38:32.2769617Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-10-10T00:38:32.7482190Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T00:38:32.7482559Z Login Succeeded 2025-10-10T00:38:32.7482998Z Configure a credential helper to remove this warning. See 2025-10-10T00:38:32.7483400Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T00:38:32.7483643Z 2025-10-10T00:38:32.7569196Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-10-10T00:38:32.7569495Z tag=${ECR_DOCKER_IMAGE##*:} 2025-10-10T00:38:32.7569756Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-10-10T00:38:32.7574513Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:32.7574757Z env: 2025-10-10T00:38:32.7574920Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:32.7575453Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:32.7575976Z ##[endgroup] 2025-10-10T00:38:32.7600131Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:32.7645470Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-10-10T00:38:32.7645773Z with: 2025-10-10T00:38:32.7646320Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:32.7647087Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:32.7647355Z env: 2025-10-10T00:38:32.7647547Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:32.7647769Z ##[endgroup] 2025-10-10T00:38:32.7668481Z ##[group]Run set -x 2025-10-10T00:38:32.7668703Z set -x 2025-10-10T00:38:32.7668878Z set +e 2025-10-10T00:38:32.7669042Z  2025-10-10T00:38:32.7669203Z login() { 2025-10-10T00:38:32.7669545Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-10-10T00:38:32.7669891Z } 2025-10-10T00:38:32.7670053Z  2025-10-10T00:38:32.7670249Z retry () { 2025-10-10T00:38:32.7670452Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-10-10T00:38:32.7670671Z } 2025-10-10T00:38:32.7670826Z  2025-10-10T00:38:32.7670999Z retry login "${DOCKER_REGISTRY}" 2025-10-10T00:38:32.7671225Z  2025-10-10T00:38:32.7671546Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-10-10T00:38:32.7671980Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-10-10T00:38:32.7672235Z  2025-10-10T00:38:32.7672390Z set -e 2025-10-10T00:38:32.7672646Z # ignore output since only exit code is used for conditional 2025-10-10T00:38:32.7673024Z # only pull docker image if it's not available locally 2025-10-10T00:38:32.7673381Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-10-10T00:38:32.7673717Z  retry docker pull "${DOCKER_IMAGE}" 2025-10-10T00:38:32.7673944Z fi 2025-10-10T00:38:32.7678432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:38:32.7678693Z env: 2025-10-10T00:38:32.7678877Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:38:32.7679437Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:32.7680081Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:32.7680385Z ##[endgroup] 2025-10-10T00:38:32.7700662Z + set +e 2025-10-10T00:38:32.7701134Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:32.7701627Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:32.7703963Z + aws ecr get-login-password --region us-east-1 2025-10-10T00:38:32.7704634Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-10-10T00:38:33.1948819Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-10-10T00:38:33.1949427Z Configure a credential helper to remove this warning. See 2025-10-10T00:38:33.1949727Z Login Succeeded 2025-10-10T00:38:33.1955955Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-10-10T00:38:33.1961193Z 2025-10-10T00:38:33.1977853Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:33.1979610Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-10-10T00:38:33.4110824Z + IMAGE_SIZE=28760.563641548157 2025-10-10T00:38:33.4111206Z + echo 'Compressed size of image in MB: 28760.563641548157' 2025-10-10T00:38:33.4111481Z + set -e 2025-10-10T00:38:33.4112482Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:33.4114191Z Compressed size of image in MB: 28760.563641548157 2025-10-10T00:38:33.4251288Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:33.4253981Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:38:33.6890313Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac: Pulling from pytorch/ci-image 2025-10-10T00:38:33.6890948Z 828c1365039a: Pulling fs layer 2025-10-10T00:38:33.6891184Z d48484dff770: Pulling fs layer 2025-10-10T00:38:33.6891393Z 63f2ff1b73ca: Pulling fs layer 2025-10-10T00:38:33.6891603Z 4a952c1cd3dd: Pulling fs layer 2025-10-10T00:38:33.6891806Z cf3649f6486a: Pulling fs layer 2025-10-10T00:38:33.6892012Z 7a4d1eb936e8: Pulling fs layer 2025-10-10T00:38:33.6892248Z fec720215749: Pulling fs layer 2025-10-10T00:38:33.6892455Z 130af8bfbe27: Pulling fs layer 2025-10-10T00:38:33.6892654Z 193b2b1a9925: Pulling fs layer 2025-10-10T00:38:33.6892853Z 38d02879e15d: Pulling fs layer 2025-10-10T00:38:33.6893059Z fb4547453b1c: Pulling fs layer 2025-10-10T00:38:33.6893273Z 5309e360108e: Pulling fs layer 2025-10-10T00:38:33.6893480Z 9be330fbd7be: Pulling fs layer 2025-10-10T00:38:33.6893685Z 2cf6366f7d11: Pulling fs layer 2025-10-10T00:38:33.6893896Z 4f4fb700ef54: Pulling fs layer 2025-10-10T00:38:33.6894099Z 56c4f51a0131: Pulling fs layer 2025-10-10T00:38:33.6894311Z f7b7f9614412: Pulling fs layer 2025-10-10T00:38:33.6894516Z 70aeb425e04c: Pulling fs layer 2025-10-10T00:38:33.6894720Z 97346936eb30: Pulling fs layer 2025-10-10T00:38:33.6894917Z 9e5860b535d4: Pulling fs layer 2025-10-10T00:38:33.6895126Z 0ba1a45e1205: Pulling fs layer 2025-10-10T00:38:33.6895329Z 3b7d78f74bbe: Pulling fs layer 2025-10-10T00:38:33.6895541Z 6beb9b78ab6e: Pulling fs layer 2025-10-10T00:38:33.6895801Z 25a0a5e5e819: Pulling fs layer 2025-10-10T00:38:33.6896001Z 5227fe91454b: Pulling fs layer 2025-10-10T00:38:33.6896197Z 66f9a01572e2: Pulling fs layer 2025-10-10T00:38:33.6896389Z c252c4979145: Pulling fs layer 2025-10-10T00:38:33.6896587Z f9c84d68915b: Pulling fs layer 2025-10-10T00:38:33.6896783Z a086fb2969ce: Pulling fs layer 2025-10-10T00:38:33.6896984Z a6c00b8f11a4: Pulling fs layer 2025-10-10T00:38:33.6897185Z fc80c2144458: Pulling fs layer 2025-10-10T00:38:33.6897377Z 3894d677ce34: Pulling fs layer 2025-10-10T00:38:33.6897579Z be6227b9675e: Pulling fs layer 2025-10-10T00:38:33.6897779Z c5731ed75d33: Pulling fs layer 2025-10-10T00:38:33.6897985Z e3d52ff40b4c: Pulling fs layer 2025-10-10T00:38:33.6898180Z 86ca23246f24: Pulling fs layer 2025-10-10T00:38:33.6898380Z 40e2321b482a: Pulling fs layer 2025-10-10T00:38:33.6898580Z b25589a402ef: Pulling fs layer 2025-10-10T00:38:33.6898781Z 3be3b4544852: Pulling fs layer 2025-10-10T00:38:33.6898972Z ca8d18c1fad6: Pulling fs layer 2025-10-10T00:38:33.6899181Z ad03627b7e8e: Pulling fs layer 2025-10-10T00:38:33.6899383Z b7b658babd8c: Pulling fs layer 2025-10-10T00:38:33.6899584Z 784eefe614e4: Pulling fs layer 2025-10-10T00:38:33.6899785Z 9084ec9ef863: Pulling fs layer 2025-10-10T00:38:33.6900017Z 0a890a7dda70: Pulling fs layer 2025-10-10T00:38:33.6900224Z 5907e244ca85: Pulling fs layer 2025-10-10T00:38:33.6900431Z 7d9cc7db846b: Pulling fs layer 2025-10-10T00:38:33.6900633Z c6f4775b7985: Pulling fs layer 2025-10-10T00:38:33.6900838Z ed36b9de7242: Pulling fs layer 2025-10-10T00:38:33.6901049Z cfc72d1580d6: Pulling fs layer 2025-10-10T00:38:33.6901245Z 140888b3776d: Pulling fs layer 2025-10-10T00:38:33.6901434Z 43ee72be7eaa: Pulling fs layer 2025-10-10T00:38:33.6901635Z 4b724c5636ce: Pulling fs layer 2025-10-10T00:38:33.6901840Z 47122b9b853f: Pulling fs layer 2025-10-10T00:38:33.6902047Z 501ccd6f6912: Pulling fs layer 2025-10-10T00:38:33.6902242Z 7c29f290a530: Pulling fs layer 2025-10-10T00:38:33.6902442Z 6f129b7d3f62: Pulling fs layer 2025-10-10T00:38:33.6902927Z f4f4042795ea: Pulling fs layer 2025-10-10T00:38:33.6903144Z a3c70cd222f0: Pulling fs layer 2025-10-10T00:38:33.6903346Z 0d733ca581b2: Pulling fs layer 2025-10-10T00:38:33.6903552Z 23e94f3ab9a7: Pulling fs layer 2025-10-10T00:38:33.6903761Z ee028affbc09: Pulling fs layer 2025-10-10T00:38:33.6904069Z f090f773ba0e: Pulling fs layer 2025-10-10T00:38:33.6904267Z 56f1f8e7f247: Pulling fs layer 2025-10-10T00:38:33.6904467Z be6227b9675e: Waiting 2025-10-10T00:38:33.6904655Z c5731ed75d33: Waiting 2025-10-10T00:38:33.6904838Z e3d52ff40b4c: Waiting 2025-10-10T00:38:33.6905012Z 86ca23246f24: Waiting 2025-10-10T00:38:33.6905192Z 4a952c1cd3dd: Waiting 2025-10-10T00:38:33.6905392Z 40e2321b482a: Waiting 2025-10-10T00:38:33.6905567Z b25589a402ef: Waiting 2025-10-10T00:38:33.6905734Z 3be3b4544852: Waiting 2025-10-10T00:38:33.6905910Z ca8d18c1fad6: Waiting 2025-10-10T00:38:33.6906087Z ad03627b7e8e: Waiting 2025-10-10T00:38:33.6906446Z b7b658babd8c: Waiting 2025-10-10T00:38:33.6906621Z 784eefe614e4: Waiting 2025-10-10T00:38:33.6906806Z cf3649f6486a: Waiting 2025-10-10T00:38:33.6906986Z 9084ec9ef863: Waiting 2025-10-10T00:38:33.6907167Z 7a4d1eb936e8: Waiting 2025-10-10T00:38:33.6907340Z fec720215749: Waiting 2025-10-10T00:38:33.6907520Z 193b2b1a9925: Waiting 2025-10-10T00:38:33.6907702Z 130af8bfbe27: Waiting 2025-10-10T00:38:33.6907888Z 5309e360108e: Waiting 2025-10-10T00:38:33.6908059Z fb4547453b1c: Waiting 2025-10-10T00:38:33.6908240Z 38d02879e15d: Waiting 2025-10-10T00:38:33.6908415Z 2cf6366f7d11: Waiting 2025-10-10T00:38:33.6908591Z 0a890a7dda70: Waiting 2025-10-10T00:38:33.6908760Z 56c4f51a0131: Waiting 2025-10-10T00:38:33.6908938Z 5907e244ca85: Waiting 2025-10-10T00:38:33.6909114Z f7b7f9614412: Waiting 2025-10-10T00:38:33.6909289Z 70aeb425e04c: Waiting 2025-10-10T00:38:33.6909461Z 9be330fbd7be: Waiting 2025-10-10T00:38:33.6909636Z 97346936eb30: Waiting 2025-10-10T00:38:33.6909813Z 9e5860b535d4: Waiting 2025-10-10T00:38:33.6909989Z 7d9cc7db846b: Waiting 2025-10-10T00:38:33.6910158Z 0ba1a45e1205: Waiting 2025-10-10T00:38:33.6910338Z 6beb9b78ab6e: Waiting 2025-10-10T00:38:33.6910519Z 3b7d78f74bbe: Waiting 2025-10-10T00:38:33.6910688Z 25a0a5e5e819: Waiting 2025-10-10T00:38:33.6910867Z c6f4775b7985: Waiting 2025-10-10T00:38:33.6911043Z 66f9a01572e2: Waiting 2025-10-10T00:38:33.6911220Z ed36b9de7242: Waiting 2025-10-10T00:38:33.6911393Z f9c84d68915b: Waiting 2025-10-10T00:38:33.6911572Z a086fb2969ce: Waiting 2025-10-10T00:38:33.6911749Z 5227fe91454b: Waiting 2025-10-10T00:38:33.6911927Z a6c00b8f11a4: Waiting 2025-10-10T00:38:33.6912095Z fc80c2144458: Waiting 2025-10-10T00:38:33.6912270Z 3894d677ce34: Waiting 2025-10-10T00:38:33.6912443Z a3c70cd222f0: Waiting 2025-10-10T00:38:33.6912621Z 43ee72be7eaa: Waiting 2025-10-10T00:38:33.6912787Z 140888b3776d: Waiting 2025-10-10T00:38:33.6912960Z 0d733ca581b2: Waiting 2025-10-10T00:38:33.6913134Z 23e94f3ab9a7: Waiting 2025-10-10T00:38:33.6913308Z ee028affbc09: Waiting 2025-10-10T00:38:33.6913474Z 4b724c5636ce: Waiting 2025-10-10T00:38:33.6913650Z 501ccd6f6912: Waiting 2025-10-10T00:38:33.6913826Z 7c29f290a530: Waiting 2025-10-10T00:38:33.6913997Z 6f129b7d3f62: Waiting 2025-10-10T00:38:33.6914165Z 47122b9b853f: Waiting 2025-10-10T00:38:33.6914338Z 56f1f8e7f247: Waiting 2025-10-10T00:38:33.6914515Z f4f4042795ea: Waiting 2025-10-10T00:38:33.6914693Z f090f773ba0e: Waiting 2025-10-10T00:38:33.6914870Z 4f4fb700ef54: Waiting 2025-10-10T00:38:33.6915047Z c252c4979145: Waiting 2025-10-10T00:38:33.6915223Z cfc72d1580d6: Waiting 2025-10-10T00:38:33.7743296Z d48484dff770: Download complete 2025-10-10T00:38:33.8637562Z 4a952c1cd3dd: Verifying Checksum 2025-10-10T00:38:33.8641542Z 4a952c1cd3dd: Download complete 2025-10-10T00:38:33.9894966Z cf3649f6486a: Verifying Checksum 2025-10-10T00:38:33.9895288Z cf3649f6486a: Download complete 2025-10-10T00:38:34.0603969Z 828c1365039a: Verifying Checksum 2025-10-10T00:38:34.0604287Z 828c1365039a: Download complete 2025-10-10T00:38:34.0900911Z 7a4d1eb936e8: Download complete 2025-10-10T00:38:34.1647889Z 130af8bfbe27: Verifying Checksum 2025-10-10T00:38:34.1648502Z 130af8bfbe27: Download complete 2025-10-10T00:38:34.2544529Z 38d02879e15d: Verifying Checksum 2025-10-10T00:38:34.2544849Z 38d02879e15d: Download complete 2025-10-10T00:38:34.3345322Z fb4547453b1c: Download complete 2025-10-10T00:38:34.4086728Z 5309e360108e: Verifying Checksum 2025-10-10T00:38:34.4087460Z 5309e360108e: Download complete 2025-10-10T00:38:34.4751553Z 9be330fbd7be: Download complete 2025-10-10T00:38:35.2803874Z 193b2b1a9925: Verifying Checksum 2025-10-10T00:38:35.2804244Z 193b2b1a9925: Download complete 2025-10-10T00:38:35.2863309Z 4f4fb700ef54: Download complete 2025-10-10T00:38:35.3168392Z 828c1365039a: Pull complete 2025-10-10T00:38:35.3316879Z d48484dff770: Pull complete 2025-10-10T00:38:35.3781270Z 56c4f51a0131: Verifying Checksum 2025-10-10T00:38:35.3781596Z 56c4f51a0131: Download complete 2025-10-10T00:38:35.4583007Z f7b7f9614412: Verifying Checksum 2025-10-10T00:38:35.4583324Z f7b7f9614412: Download complete 2025-10-10T00:38:35.5356409Z 70aeb425e04c: Verifying Checksum 2025-10-10T00:38:35.6447941Z 97346936eb30: Verifying Checksum 2025-10-10T00:38:35.6454070Z 97346936eb30: Download complete 2025-10-10T00:38:35.7224264Z 9e5860b535d4: Verifying Checksum 2025-10-10T00:38:35.7224739Z 9e5860b535d4: Download complete 2025-10-10T00:38:35.8076539Z 0ba1a45e1205: Verifying Checksum 2025-10-10T00:38:35.8077070Z 0ba1a45e1205: Download complete 2025-10-10T00:38:35.9102447Z 3b7d78f74bbe: Verifying Checksum 2025-10-10T00:38:35.9102764Z 3b7d78f74bbe: Download complete 2025-10-10T00:38:35.9884127Z 6beb9b78ab6e: Verifying Checksum 2025-10-10T00:38:35.9884653Z 6beb9b78ab6e: Download complete 2025-10-10T00:38:36.0824415Z 25a0a5e5e819: Verifying Checksum 2025-10-10T00:38:36.0824752Z 25a0a5e5e819: Download complete 2025-10-10T00:38:36.1779800Z 5227fe91454b: Verifying Checksum 2025-10-10T00:38:36.1781987Z 5227fe91454b: Download complete 2025-10-10T00:38:36.2649029Z 66f9a01572e2: Verifying Checksum 2025-10-10T00:38:36.2653411Z 66f9a01572e2: Download complete 2025-10-10T00:38:36.3556838Z c252c4979145: Verifying Checksum 2025-10-10T00:38:36.3557171Z c252c4979145: Download complete 2025-10-10T00:38:36.8894526Z 63f2ff1b73ca: Verifying Checksum 2025-10-10T00:38:36.8899721Z 63f2ff1b73ca: Download complete 2025-10-10T00:38:36.9693382Z a086fb2969ce: Verifying Checksum 2025-10-10T00:38:36.9693695Z a086fb2969ce: Download complete 2025-10-10T00:38:37.0410043Z a6c00b8f11a4: Verifying Checksum 2025-10-10T00:38:37.0410506Z a6c00b8f11a4: Download complete 2025-10-10T00:38:37.1077024Z fc80c2144458: Download complete 2025-10-10T00:38:37.1995164Z 3894d677ce34: Download complete 2025-10-10T00:38:37.4530721Z be6227b9675e: Verifying Checksum 2025-10-10T00:38:37.4531029Z be6227b9675e: Download complete 2025-10-10T00:38:37.5511650Z c5731ed75d33: Download complete 2025-10-10T00:38:37.6395093Z e3d52ff40b4c: Verifying Checksum 2025-10-10T00:38:37.6398703Z e3d52ff40b4c: Download complete 2025-10-10T00:38:37.7299208Z 86ca23246f24: Verifying Checksum 2025-10-10T00:38:37.7299528Z 86ca23246f24: Download complete 2025-10-10T00:38:37.8013034Z 40e2321b482a: Verifying Checksum 2025-10-10T00:38:37.8013517Z 40e2321b482a: Download complete 2025-10-10T00:38:37.8943780Z b25589a402ef: Download complete 2025-10-10T00:38:41.0176978Z f9c84d68915b: Verifying Checksum 2025-10-10T00:38:41.0177273Z f9c84d68915b: Download complete 2025-10-10T00:38:41.1160305Z ca8d18c1fad6: Verifying Checksum 2025-10-10T00:38:41.1160844Z ca8d18c1fad6: Download complete 2025-10-10T00:38:44.3069197Z ad03627b7e8e: Verifying Checksum 2025-10-10T00:38:44.3069536Z ad03627b7e8e: Download complete 2025-10-10T00:38:47.9549872Z 63f2ff1b73ca: Pull complete 2025-10-10T00:38:48.2371048Z 4a952c1cd3dd: Pull complete 2025-10-10T00:38:48.6304050Z cf3649f6486a: Pull complete 2025-10-10T00:38:49.1311989Z 7a4d1eb936e8: Pull complete 2025-10-10T00:38:49.4372607Z fec720215749: Pull complete 2025-10-10T00:38:49.8707960Z 130af8bfbe27: Pull complete 2025-10-10T00:38:53.5667407Z 193b2b1a9925: Pull complete 2025-10-10T00:38:53.8415669Z 38d02879e15d: Pull complete 2025-10-10T00:38:54.0840858Z fb4547453b1c: Pull complete 2025-10-10T00:38:54.4364859Z 5309e360108e: Pull complete 2025-10-10T00:38:54.7374398Z 9be330fbd7be: Pull complete 2025-10-10T00:39:08.7551854Z 2cf6366f7d11: Download complete 2025-10-10T00:39:08.8558203Z 784eefe614e4: Verifying Checksum 2025-10-10T00:39:08.8558496Z 784eefe614e4: Download complete 2025-10-10T00:39:08.9465949Z 9084ec9ef863: Download complete 2025-10-10T00:39:09.0330548Z 0a890a7dda70: Verifying Checksum 2025-10-10T00:39:09.0330989Z 0a890a7dda70: Download complete 2025-10-10T00:39:09.1204917Z 5907e244ca85: Download complete 2025-10-10T00:39:09.2100352Z 7d9cc7db846b: Download complete 2025-10-10T00:39:09.3016708Z c6f4775b7985: Verifying Checksum 2025-10-10T00:39:09.3018756Z c6f4775b7985: Download complete 2025-10-10T00:39:09.3828105Z ed36b9de7242: Verifying Checksum 2025-10-10T00:39:09.3828425Z ed36b9de7242: Download complete 2025-10-10T00:39:09.4668085Z cfc72d1580d6: Download complete 2025-10-10T00:39:09.5561740Z 140888b3776d: Verifying Checksum 2025-10-10T00:39:09.5564990Z 140888b3776d: Download complete 2025-10-10T00:39:09.6521148Z 43ee72be7eaa: Verifying Checksum 2025-10-10T00:39:09.6521471Z 43ee72be7eaa: Download complete 2025-10-10T00:39:09.7292960Z 4b724c5636ce: Verifying Checksum 2025-10-10T00:39:09.7293288Z 4b724c5636ce: Download complete 2025-10-10T00:39:09.8385502Z 47122b9b853f: Verifying Checksum 2025-10-10T00:39:09.8386034Z 47122b9b853f: Download complete 2025-10-10T00:39:09.9200475Z 501ccd6f6912: Download complete 2025-10-10T00:39:10.0081448Z 7c29f290a530: Download complete 2025-10-10T00:39:12.5205545Z 6f129b7d3f62: Verifying Checksum 2025-10-10T00:39:12.5215620Z 6f129b7d3f62: Download complete 2025-10-10T00:39:12.6162697Z f4f4042795ea: Download complete 2025-10-10T00:39:12.7127071Z a3c70cd222f0: Verifying Checksum 2025-10-10T00:39:12.7127392Z a3c70cd222f0: Download complete 2025-10-10T00:39:12.7836350Z 0d733ca581b2: Download complete 2025-10-10T00:39:12.8767583Z 23e94f3ab9a7: Verifying Checksum 2025-10-10T00:39:12.8767880Z 23e94f3ab9a7: Download complete 2025-10-10T00:39:12.9564225Z ee028affbc09: Verifying Checksum 2025-10-10T00:39:12.9564674Z ee028affbc09: Download complete 2025-10-10T00:39:13.0410410Z f090f773ba0e: Verifying Checksum 2025-10-10T00:39:13.0410704Z f090f773ba0e: Download complete 2025-10-10T00:39:13.6726351Z 56f1f8e7f247: Verifying Checksum 2025-10-10T00:39:13.6726706Z 56f1f8e7f247: Download complete 2025-10-10T00:39:49.7252679Z b7b658babd8c: Verifying Checksum 2025-10-10T00:39:49.7252982Z b7b658babd8c: Download complete 2025-10-10T00:40:28.9684115Z 2cf6366f7d11: Pull complete 2025-10-10T00:40:29.2615862Z 4f4fb700ef54: Pull complete 2025-10-10T00:40:29.5765606Z 56c4f51a0131: Pull complete 2025-10-10T00:40:29.8896269Z f7b7f9614412: Pull complete 2025-10-10T00:40:30.1694850Z 70aeb425e04c: Pull complete 2025-10-10T00:40:30.7020807Z 97346936eb30: Pull complete 2025-10-10T00:40:31.1512472Z 9e5860b535d4: Pull complete 2025-10-10T00:40:31.4727098Z 0ba1a45e1205: Pull complete 2025-10-10T00:40:31.8093906Z 3b7d78f74bbe: Pull complete 2025-10-10T00:40:32.2645240Z 6beb9b78ab6e: Pull complete 2025-10-10T00:40:32.7187305Z 25a0a5e5e819: Pull complete 2025-10-10T00:40:33.2136900Z 5227fe91454b: Pull complete 2025-10-10T00:40:34.2002067Z 66f9a01572e2: Pull complete 2025-10-10T00:40:34.7314765Z c252c4979145: Pull complete 2025-10-10T00:40:46.4720826Z f9c84d68915b: Pull complete 2025-10-10T00:40:46.8851931Z a086fb2969ce: Pull complete 2025-10-10T00:40:47.3530415Z a6c00b8f11a4: Pull complete 2025-10-10T00:40:48.1577230Z fc80c2144458: Pull complete 2025-10-10T00:40:48.5063109Z 3894d677ce34: Pull complete 2025-10-10T00:40:49.3314941Z be6227b9675e: Pull complete 2025-10-10T00:40:49.8583661Z c5731ed75d33: Pull complete 2025-10-10T00:40:50.3062381Z e3d52ff40b4c: Pull complete 2025-10-10T00:40:51.0608628Z 86ca23246f24: Pull complete 2025-10-10T00:40:51.5416229Z 40e2321b482a: Pull complete 2025-10-10T00:40:51.8725788Z b25589a402ef: Pull complete 2025-10-10T00:41:44.4978953Z 3be3b4544852: Verifying Checksum 2025-10-10T00:41:44.4979478Z 3be3b4544852: Download complete 2025-10-10T00:46:03.9321164Z 3be3b4544852: Pull complete 2025-10-10T00:46:03.9606330Z ca8d18c1fad6: Pull complete 2025-10-10T00:46:06.8744900Z ad03627b7e8e: Pull complete 2025-10-10T00:48:42.2327026Z b7b658babd8c: Pull complete 2025-10-10T00:48:42.2650800Z 784eefe614e4: Pull complete 2025-10-10T00:48:42.2901623Z 9084ec9ef863: Pull complete 2025-10-10T00:48:42.3396483Z 0a890a7dda70: Pull complete 2025-10-10T00:48:42.3977791Z 5907e244ca85: Pull complete 2025-10-10T00:48:42.4237221Z 7d9cc7db846b: Pull complete 2025-10-10T00:48:42.4801680Z c6f4775b7985: Pull complete 2025-10-10T00:48:42.5316942Z ed36b9de7242: Pull complete 2025-10-10T00:48:42.5602048Z cfc72d1580d6: Pull complete 2025-10-10T00:48:42.6134541Z 140888b3776d: Pull complete 2025-10-10T00:48:42.6402677Z 43ee72be7eaa: Pull complete 2025-10-10T00:48:42.6931655Z 4b724c5636ce: Pull complete 2025-10-10T00:48:42.7222795Z 47122b9b853f: Pull complete 2025-10-10T00:48:42.7775994Z 501ccd6f6912: Pull complete 2025-10-10T00:48:42.8028972Z 7c29f290a530: Pull complete 2025-10-10T00:48:52.9490956Z 6f129b7d3f62: Pull complete 2025-10-10T00:48:53.3818213Z f4f4042795ea: Pull complete 2025-10-10T00:48:53.7919734Z a3c70cd222f0: Pull complete 2025-10-10T00:48:54.1954573Z 0d733ca581b2: Pull complete 2025-10-10T00:48:54.5958480Z 23e94f3ab9a7: Pull complete 2025-10-10T00:48:55.1133453Z ee028affbc09: Pull complete 2025-10-10T00:48:56.0165408Z f090f773ba0e: Pull complete 2025-10-10T00:48:58.7186246Z 56f1f8e7f247: Pull complete 2025-10-10T00:48:59.4299984Z Digest: sha256:2e6e7297c9105722429dc16c6731f2c79bd9d6b81eb7a39feebebd90cfe1676f 2025-10-10T00:48:59.5048270Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:48:59.5510312Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:48:59.5564748Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:48:59.5565377Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-10-10T00:48:59.5572827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:48:59.5573108Z env: 2025-10-10T00:48:59.5573284Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:48:59.5573487Z ##[endgroup] 2025-10-10T00:48:59.5650770Z Prepare all required actions 2025-10-10T00:48:59.5902519Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-10-10T00:48:59.5902791Z with: 2025-10-10T00:48:59.5903488Z github-token: *** 2025-10-10T00:48:59.5903667Z env: 2025-10-10T00:48:59.5903849Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:48:59.5904057Z ##[endgroup] 2025-10-10T00:48:59.6049373Z ##[group]Run set -eux 2025-10-10T00:48:59.6049592Z set -eux 2025-10-10T00:48:59.6049889Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-10-10T00:48:59.6055293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:48:59.6055537Z env: 2025-10-10T00:48:59.6055700Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:48:59.6056391Z GITHUB_TOKEN: *** 2025-10-10T00:48:59.6056573Z ##[endgroup] 2025-10-10T00:48:59.6080220Z + python3 .github/scripts/get_workflow_job_id.py 18392306145 i-01dad1d3a0e80f7c0 2025-10-10T00:49:00.3499632Z Setting output job-id=52406535390 2025-10-10T00:49:00.3500237Z Setting output job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:00.3779599Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-10-10T00:49:00.3780119Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-10-10T00:49:00.3780745Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-10-10T00:49:00.3781329Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-10-10T00:49:00.3787814Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:00.3788077Z env: 2025-10-10T00:49:00.3788258Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:00.3788590Z JOB_ID: 52406535390 2025-10-10T00:49:00.3788913Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:00.3789263Z WORKFLOW_NAME: inductor 2025-10-10T00:49:00.3789473Z WORKFLOW_RUN_ID: 18392306145 2025-10-10T00:49:00.3789731Z MONITOR_LOG_INTERVAL: 5 2025-10-10T00:49:00.3789943Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-10-10T00:49:00.3790165Z ##[endgroup] 2025-10-10T00:49:01.2667516Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T00:49:01.5822017Z Collecting psutil==5.9.8 2025-10-10T00:49:01.6002492Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-10-10T00:49:01.7247611Z Collecting dataclasses_json==0.6.7 2025-10-10T00:49:01.7297679Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-10-10T00:49:01.7870946Z Collecting nvidia-ml-py==11.525.84 2025-10-10T00:49:01.7994043Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-10-10T00:49:01.8527998Z Collecting typing-inspect<1,>=0.4.0 2025-10-10T00:49:01.8561527Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-10-10T00:49:02.0130919Z Collecting marshmallow<4.0.0,>=3.18.0 2025-10-10T00:49:02.0171048Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-10-10T00:49:02.1870638Z Collecting packaging>=17.0 2025-10-10T00:49:02.2005109Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-10-10T00:49:02.3726251Z Collecting typing-extensions>=3.7.4 2025-10-10T00:49:02.3764148Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-10-10T00:49:02.4965982Z Collecting mypy-extensions>=0.3.0 2025-10-10T00:49:02.5002561Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-10-10T00:49:02.7970029Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-10-10T00:49:03.5290016Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-10-10T00:49:03.7651095Z Prepare all required actions 2025-10-10T00:49:03.7651405Z Getting action download info 2025-10-10T00:49:03.9230458Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-10-10T00:49:04.2402417Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-10-10T00:49:05.6957710Z ##[group]Run ./.github/actions/download-build-artifacts 2025-10-10T00:49:05.6957998Z with: 2025-10-10T00:49:05.6958202Z name: linux-jammy-py3.10-gcc11-build 2025-10-10T00:49:05.6958449Z s3-bucket: gha-artifacts 2025-10-10T00:49:05.6958646Z env: 2025-10-10T00:49:05.6958814Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:05.6959004Z ##[endgroup] 2025-10-10T00:49:05.6991373Z ##[group]Run seemethere/download-artifact-s3@v4 2025-10-10T00:49:05.6991646Z with: 2025-10-10T00:49:05.6991839Z name: linux-jammy-py3.10-gcc11-build 2025-10-10T00:49:05.6992077Z s3-bucket: gha-artifacts 2025-10-10T00:49:05.6992332Z region: us-east-1 2025-10-10T00:49:05.6992499Z env: 2025-10-10T00:49:05.6992665Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:05.6992853Z ##[endgroup] 2025-10-10T00:49:06.0998797Z (node:48420) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-10-10T00:49:06.0999352Z 2025-10-10T00:49:06.1004020Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-10-10T00:49:06.1010165Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-10-10T00:49:06.1015366Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-10-10T00:49:06.3906813Z Found 1 objects with prefix pytorch/pytorch/18392306145/linux-jammy-py3.10-gcc11-build/ 2025-10-10T00:49:06.3907401Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-10-10T00:49:12.5209144Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-10-10T00:49:12.5210418Z Artifact download has finished successfully 2025-10-10T00:49:12.5543221Z ##[group]Run unzip -o artifacts.zip 2025-10-10T00:49:12.5543500Z unzip -o artifacts.zip 2025-10-10T00:49:12.5549735Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:12.5550017Z env: 2025-10-10T00:49:12.5550198Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:12.5550397Z ##[endgroup] 2025-10-10T00:49:12.6079588Z Archive: artifacts.zip 2025-10-10T00:49:12.6079985Z creating: dist/ 2025-10-10T00:49:13.7351809Z inflating: dist/torch-2.10.0a0+git344e636-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:13.7352179Z creating: dist/vision/ 2025-10-10T00:49:13.7433251Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:13.7433665Z creating: dist/audio/ 2025-10-10T00:49:13.7463475Z inflating: dist/audio/torchaudio-2.8.0a0+87ff22e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:13.7463870Z creating: dist/ao/ 2025-10-10T00:49:13.7506482Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-10-10T00:49:13.7624213Z inflating: dist/.ninja_log 2025-10-10T00:49:13.7624746Z creating: build/custom_test_artifacts/ 2025-10-10T00:49:13.7625157Z creating: build/custom_test_artifacts/custom-op-build/ 2025-10-10T00:49:13.7625657Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-10-10T00:49:13.7626872Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-10-10T00:49:13.7627668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T00:49:13.7628247Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-10-10T00:49:13.7631229Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T00:49:13.7631965Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T00:49:13.7632898Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T00:49:13.7638649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T00:49:13.7643690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T00:49:13.7649336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T00:49:13.7651473Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T00:49:13.7651950Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T00:49:13.7652457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T00:49:13.7652967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T00:49:13.7653551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T00:49:13.7654088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T00:49:13.7654663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T00:49:13.7655171Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-10-10T00:49:13.7655574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-10-10T00:49:13.7655985Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-10-10T00:49:13.7656434Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-10-10T00:49:13.7656938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-10-10T00:49:13.7657682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-10-10T00:49:13.7658122Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-10-10T00:49:13.7658588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-10-10T00:49:13.7659056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-10-10T00:49:13.7659521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-10-10T00:49:13.7659986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-10-10T00:49:13.7660472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-10-10T00:49:13.7662200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-10-10T00:49:13.7851872Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-10-10T00:49:13.7852369Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-10-10T00:49:13.7852867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-10-10T00:49:13.7853517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-10-10T00:49:13.7854018Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-10-10T00:49:13.7854481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-10-10T00:49:13.7854967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-10-10T00:49:13.7855452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-10-10T00:49:13.7856250Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-10-10T00:49:13.7856738Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-10-10T00:49:13.7857216Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-10-10T00:49:13.7874692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-10-10T00:49:13.7954165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-10-10T00:49:13.7955770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T00:49:13.7956325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-10-10T00:49:13.7956818Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-10-10T00:49:13.7957251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-10-10T00:49:13.7957670Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-10-10T00:49:13.7958093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-10-10T00:49:13.7958462Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-10-10T00:49:13.7958838Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-10-10T00:49:13.8121620Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-10-10T00:49:13.8176645Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-10-10T00:49:13.8177062Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-10-10T00:49:13.8177399Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-10-10T00:49:13.8178124Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-10-10T00:49:13.8179683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T00:49:13.8180123Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-10-10T00:49:13.8180547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T00:49:13.8180997Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T00:49:13.8181431Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T00:49:13.8183714Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T00:49:13.8184355Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T00:49:13.8184840Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T00:49:13.8185402Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T00:49:13.8185867Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T00:49:13.8187110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T00:49:13.8188609Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T00:49:13.8189132Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T00:49:13.8191642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T00:49:13.8192314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T00:49:13.8198323Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-10-10T00:49:13.8203752Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-10-10T00:49:13.8205326Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-10-10T00:49:13.8205868Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-10-10T00:49:13.8212115Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-10-10T00:49:13.8214351Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-10-10T00:49:13.8215048Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-10-10T00:49:13.8221121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-10-10T00:49:13.8224260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-10-10T00:49:13.8224872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-10-10T00:49:13.8225416Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-10-10T00:49:13.8225933Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-10-10T00:49:13.8226580Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-10-10T00:49:13.8275542Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-10-10T00:49:13.8282097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T00:49:13.8282835Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-10-10T00:49:13.8283316Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-10-10T00:49:13.8284453Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-10-10T00:49:13.8285389Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-10-10T00:49:13.8285842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-10-10T00:49:13.8286221Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-10-10T00:49:13.8286591Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-10-10T00:49:13.8318034Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-10-10T00:49:13.8318450Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-10-10T00:49:13.8319024Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-10-10T00:49:13.8319466Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-10-10T00:49:13.8322468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-10-10T00:49:13.8323221Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-10-10T00:49:13.8323710Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-10-10T00:49:13.8324207Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-10-10T00:49:13.8324712Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-10-10T00:49:13.8325411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-10-10T00:49:13.8325996Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-10-10T00:49:13.8326522Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-10-10T00:49:13.8327043Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-10-10T00:49:13.8327778Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-10-10T00:49:13.8333237Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-10-10T00:49:13.8335935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-10-10T00:49:13.8336498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-10-10T00:49:13.8337081Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-10-10T00:49:13.8337719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-10-10T00:49:13.8338281Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-10-10T00:49:13.8338764Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-10-10T00:49:13.8339249Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-10-10T00:49:13.8339780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-10-10T00:49:13.8340365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-10-10T00:49:13.8340938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-10-10T00:49:13.8341448Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-10-10T00:49:13.8342000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-10-10T00:49:13.8342548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-10-10T00:49:13.8343350Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-10-10T00:49:13.8343933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-10-10T00:49:13.8344519Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-10-10T00:49:13.8345119Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-10-10T00:49:13.8454717Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-10-10T00:49:13.8457405Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-10-10T00:49:13.8457992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-10-10T00:49:13.8458632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-10-10T00:49:13.8459190Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-10-10T00:49:13.8459713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-10-10T00:49:13.8460246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-10-10T00:49:13.8460795Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-10-10T00:49:13.8461378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-10-10T00:49:13.8461981Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-10-10T00:49:13.8462572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-10-10T00:49:13.8481942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-10-10T00:49:13.8526632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-10-10T00:49:13.8528826Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-10-10T00:49:13.8529345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-10-10T00:49:13.8529810Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-10-10T00:49:13.8530253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-10-10T00:49:13.8530689Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-10-10T00:49:13.8531131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-10-10T00:49:13.8534109Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-10-10T00:49:13.8538062Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-10-10T00:49:13.8627739Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-10-10T00:49:13.8665933Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-10-10T00:49:13.8666526Z creating: build/lib/ 2025-10-10T00:49:13.8750715Z inflating: build/lib/libprotobuf-lite.a 2025-10-10T00:49:13.9171884Z inflating: build/lib/libprotobuf.a 2025-10-10T00:49:13.9657016Z inflating: build/lib/libprotoc.a 2025-10-10T00:49:13.9665852Z inflating: build/lib/libpthreadpool.a 2025-10-10T00:49:13.9675525Z inflating: build/lib/libcpuinfo.a 2025-10-10T00:49:13.9681219Z inflating: build/lib/libcpuinfo_internals.a 2025-10-10T00:49:13.9681846Z inflating: build/lib/libclog.a 2025-10-10T00:49:13.9700669Z inflating: build/lib/libpytorch_qnnpack.a 2025-10-10T00:49:13.9702593Z inflating: build/lib/libnnpack_reference_layers.a 2025-10-10T00:49:13.9886419Z inflating: build/lib/libmicrokernels-prod.a 2025-10-10T00:49:13.9902738Z inflating: build/lib/libnnpack.a 2025-10-10T00:49:14.0771025Z inflating: build/lib/libmicrokernels-all.a 2025-10-10T00:49:14.0842448Z inflating: build/lib/libgtest.a 2025-10-10T00:49:14.0858669Z inflating: build/lib/libgmock.a 2025-10-10T00:49:14.0861706Z inflating: build/lib/libgmock_main.a 2025-10-10T00:49:14.0950279Z inflating: build/lib/libXNNPACK.a 2025-10-10T00:49:14.0955023Z inflating: build/lib/libgtest_main.a 2025-10-10T00:49:14.1023133Z inflating: build/lib/libbenchmark.a 2025-10-10T00:49:14.1023491Z inflating: build/lib/libbenchmark_main.a 2025-10-10T00:49:14.1030400Z inflating: build/lib/libittnotify.a 2025-10-10T00:49:14.1030922Z inflating: build/lib/libjitprofiling.a 2025-10-10T00:49:14.1097357Z inflating: build/lib/libasmjit.a 2025-10-10T00:49:14.2206339Z inflating: build/lib/libfbgemm.a 2025-10-10T00:49:14.2236962Z inflating: build/lib/libtensorpipe_uv.a 2025-10-10T00:49:14.2772189Z inflating: build/lib/libtensorpipe.a 2025-10-10T00:49:14.2892529Z inflating: build/lib/libgloo.a 2025-10-10T00:49:14.2938966Z inflating: build/lib/libonnx_proto.a 2025-10-10T00:49:14.3633251Z inflating: build/lib/libonnx.a 2025-10-10T00:49:15.3279095Z inflating: build/lib/libdnnl.a 2025-10-10T00:49:15.3297881Z inflating: build/lib/libfmt.a 2025-10-10T00:49:15.3564565Z inflating: build/lib/libkineto.a 2025-10-10T00:49:15.3676929Z inflating: build/lib/libc10.so 2025-10-10T00:49:15.3683404Z inflating: build/lib/libtorch_global_deps.so 2025-10-10T00:49:18.2027170Z inflating: build/lib/libtorch_cpu.so 2025-10-10T00:49:18.2033839Z inflating: build/lib/libtorch.so 2025-10-10T00:49:18.2093309Z inflating: build/lib/libtorchbind_test.so 2025-10-10T00:49:18.2112150Z inflating: build/lib/libjitbackend_test.so 2025-10-10T00:49:18.2138704Z inflating: build/lib/libbackend_with_compiler.so 2025-10-10T00:49:18.2169353Z inflating: build/lib/libaoti_custom_ops.so 2025-10-10T00:49:18.2173766Z inflating: build/lib/libshm.so 2025-10-10T00:49:18.4302964Z inflating: build/lib/libtorch_python.so 2025-10-10T00:49:18.4339459Z inflating: build/lib/libnnapi_backend.so 2025-10-10T00:49:18.4339832Z creating: build/bin/ 2025-10-10T00:49:18.4340050Z creating: build/bin/CMakeFiles/ 2025-10-10T00:49:18.4340296Z inflating: build/bin/cmake_install.cmake 2025-10-10T00:49:18.4340550Z inflating: build/bin/CTestTestfile.cmake 2025-10-10T00:49:18.4797845Z inflating: build/bin/protoc-3.13.0.0 2025-10-10T00:49:18.5254905Z inflating: build/bin/protoc 2025-10-10T00:49:18.5314894Z inflating: build/bin/c10_AllocatorConfig_test 2025-10-10T00:49:18.5374090Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-10-10T00:49:18.5429580Z inflating: build/bin/c10_DeviceGuard_test 2025-10-10T00:49:18.5495303Z inflating: build/bin/c10_DispatchKeySet_test 2025-10-10T00:49:18.5553384Z inflating: build/bin/c10_Device_test 2025-10-10T00:49:18.5612601Z inflating: build/bin/c10_Scalar_test 2025-10-10T00:49:18.5667252Z inflating: build/bin/c10_StreamGuard_test 2025-10-10T00:49:18.5731852Z inflating: build/bin/c10_SymInt_test 2025-10-10T00:49:18.5796318Z inflating: build/bin/c10_SizesAndStrides_test 2025-10-10T00:49:18.5848375Z inflating: build/bin/c10_ArrayRef_test 2025-10-10T00:49:18.5907778Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-10-10T00:49:18.5971820Z inflating: build/bin/c10_InlineStreamGuard_test 2025-10-10T00:49:18.6046382Z inflating: build/bin/c10_cow_test 2025-10-10T00:49:18.6102181Z inflating: build/bin/c10_Bitset_test 2025-10-10T00:49:18.6157179Z inflating: build/bin/c10_ConstexprCrc_test 2025-10-10T00:49:18.6219016Z inflating: build/bin/c10_Enumerate_test 2025-10-10T00:49:18.6276739Z inflating: build/bin/c10_DeadlockDetection_test 2025-10-10T00:49:18.6332182Z inflating: build/bin/c10_IntrusiveList_test 2025-10-10T00:49:18.6390623Z inflating: build/bin/c10_Half_test 2025-10-10T00:49:18.6450238Z inflating: build/bin/c10_LeftRight_test 2025-10-10T00:49:18.6511517Z inflating: build/bin/c10_Metaprogramming_test 2025-10-10T00:49:18.6566673Z inflating: build/bin/c10_Semaphore_test 2025-10-10T00:49:18.6626232Z inflating: build/bin/c10_NetworkFlow_test 2025-10-10T00:49:18.6684145Z inflating: build/bin/c10_Synchronized_test 2025-10-10T00:49:18.6742726Z inflating: build/bin/c10_ThreadLocal_test 2025-10-10T00:49:18.6798543Z inflating: build/bin/c10_TypeIndex_test 2025-10-10T00:49:18.6855699Z inflating: build/bin/c10_TypeList_test 2025-10-10T00:49:18.6910042Z inflating: build/bin/c10_TypeTraits_test 2025-10-10T00:49:18.6968084Z inflating: build/bin/c10_accumulate_test 2025-10-10T00:49:18.7025663Z inflating: build/bin/c10_bfloat16_test 2025-10-10T00:49:18.7081830Z inflating: build/bin/c10_bit_cast_test 2025-10-10T00:49:18.7143850Z inflating: build/bin/c10_complex_math_test 2025-10-10T00:49:18.7202996Z inflating: build/bin/c10_complex_test 2025-10-10T00:49:18.7260808Z inflating: build/bin/c10_exception_test 2025-10-10T00:49:18.7317382Z inflating: build/bin/c10_error_test 2025-10-10T00:49:18.7371184Z inflating: build/bin/c10_flags_test 2025-10-10T00:49:18.7424953Z inflating: build/bin/c10_generic_math_test 2025-10-10T00:49:18.7486427Z inflating: build/bin/c10_lazy_test 2025-10-10T00:49:18.7541142Z inflating: build/bin/c10_irange_test 2025-10-10T00:49:18.7604268Z inflating: build/bin/c10_logging_test 2025-10-10T00:49:18.7773109Z inflating: build/bin/c10_intrusive_ptr_test 2025-10-10T00:49:18.7852551Z inflating: build/bin/c10_optional_test 2025-10-10T00:49:18.7916551Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-10-10T00:49:18.7973306Z inflating: build/bin/c10_registry_test 2025-10-10T00:49:18.8135844Z inflating: build/bin/c10_small_vector_test 2025-10-10T00:49:18.8190727Z inflating: build/bin/c10_ssize_test 2025-10-10T00:49:18.8249766Z inflating: build/bin/c10_tempfile_test 2025-10-10T00:49:18.8309226Z inflating: build/bin/c10_string_util_test 2025-10-10T00:49:18.8357119Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-10-10T00:49:18.8410831Z inflating: build/bin/c10_string_view_test 2025-10-10T00:49:18.8475117Z inflating: build/bin/c10_typeid_test 2025-10-10T00:49:18.9035952Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-10-10T00:49:18.9610998Z inflating: build/bin/vec_test_all_types_AVX512 2025-10-10T00:49:19.0190450Z inflating: build/bin/vec_test_all_types_AVX2 2025-10-10T00:49:19.0253582Z inflating: build/bin/static_runtime_bench 2025-10-10T00:49:19.0493260Z inflating: build/bin/static_runtime_test 2025-10-10T00:49:19.0569113Z inflating: build/bin/Dict_test 2025-10-10T00:49:19.0620362Z inflating: build/bin/Dimname_test 2025-10-10T00:49:19.0687948Z inflating: build/bin/MaybeOwned_test 2025-10-10T00:49:19.0750706Z inflating: build/bin/NamedTensor_test 2025-10-10T00:49:19.0811062Z inflating: build/bin/apply_utils_test 2025-10-10T00:49:19.0875476Z inflating: build/bin/atest 2025-10-10T00:49:19.0939081Z inflating: build/bin/basic 2025-10-10T00:49:19.0997091Z inflating: build/bin/broadcast_test 2025-10-10T00:49:19.1051087Z inflating: build/bin/cpu_allocator_test 2025-10-10T00:49:19.1108072Z inflating: build/bin/cpu_generator_test 2025-10-10T00:49:19.1168339Z inflating: build/bin/cpu_profiling_allocator_test 2025-10-10T00:49:19.1261224Z inflating: build/bin/cpu_rng_test 2025-10-10T00:49:19.1314045Z inflating: build/bin/dlconvertor_test 2025-10-10T00:49:19.1374433Z inflating: build/bin/extension_backend_test 2025-10-10T00:49:19.1433870Z inflating: build/bin/half_test 2025-10-10T00:49:19.1526869Z inflating: build/bin/ivalue_test 2025-10-10T00:49:19.1576545Z inflating: build/bin/lazy_tensor_test 2025-10-10T00:49:19.1631662Z inflating: build/bin/math_kernel_test 2025-10-10T00:49:19.1683769Z inflating: build/bin/memory_format_test 2025-10-10T00:49:19.1740463Z inflating: build/bin/memory_overlapping_test 2025-10-10T00:49:19.1796083Z inflating: build/bin/mobile_memory_cleanup 2025-10-10T00:49:19.1854603Z inflating: build/bin/native_test 2025-10-10T00:49:19.1907591Z inflating: build/bin/operator_name_test 2025-10-10T00:49:19.1960000Z inflating: build/bin/operators_test 2025-10-10T00:49:19.2015967Z inflating: build/bin/packedtensoraccessor_test 2025-10-10T00:49:19.2082183Z inflating: build/bin/pow_test 2025-10-10T00:49:19.2141781Z inflating: build/bin/quantized_test 2025-10-10T00:49:19.2190710Z inflating: build/bin/reduce_ops_test 2025-10-10T00:49:19.2244049Z inflating: build/bin/reportMemoryUsage_test 2025-10-10T00:49:19.2300652Z inflating: build/bin/scalar_tensor_test 2025-10-10T00:49:19.2360472Z inflating: build/bin/scalar_test 2025-10-10T00:49:19.2412731Z inflating: build/bin/StorageUtils_test 2025-10-10T00:49:19.2474106Z inflating: build/bin/stride_properties_test 2025-10-10T00:49:19.2575788Z inflating: build/bin/tensor_iterator_test 2025-10-10T00:49:19.2649642Z inflating: build/bin/test_parallel 2025-10-10T00:49:19.2718652Z inflating: build/bin/thread_init_test 2025-10-10T00:49:19.2774593Z inflating: build/bin/type_ptr_test 2025-10-10T00:49:19.2832022Z inflating: build/bin/type_test 2025-10-10T00:49:19.2888113Z inflating: build/bin/undefined_tensor_test 2025-10-10T00:49:19.2941033Z inflating: build/bin/verify_api_visibility 2025-10-10T00:49:19.3013145Z inflating: build/bin/legacy_vmap_test 2025-10-10T00:49:19.3067326Z inflating: build/bin/weakref_test 2025-10-10T00:49:19.3122103Z inflating: build/bin/wrapdim_test 2025-10-10T00:49:19.3176710Z inflating: build/bin/xla_tensor_test 2025-10-10T00:49:19.3243190Z inflating: build/bin/IListRef_test 2025-10-10T00:49:19.3345630Z inflating: build/bin/List_test 2025-10-10T00:49:19.3412818Z inflating: build/bin/KernelFunction_test 2025-10-10T00:49:19.3537086Z inflating: build/bin/kernel_function_legacy_test 2025-10-10T00:49:19.3624912Z inflating: build/bin/kernel_function_test 2025-10-10T00:49:19.3746210Z inflating: build/bin/kernel_lambda_legacy_test 2025-10-10T00:49:19.3847849Z inflating: build/bin/kernel_lambda_test 2025-10-10T00:49:19.3912822Z inflating: build/bin/kernel_stackbased_test 2025-10-10T00:49:19.4010821Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-10-10T00:49:19.4064949Z inflating: build/bin/CppSignature_test 2025-10-10T00:49:19.4117440Z inflating: build/bin/backend_fallback_test 2025-10-10T00:49:19.4172409Z inflating: build/bin/op_allowlist_test 2025-10-10T00:49:19.4468307Z inflating: build/bin/op_registration_test 2025-10-10T00:49:19.4537127Z inflating: build/bin/inline_container_test 2025-10-10T00:49:19.4594809Z inflating: build/bin/TCPStoreTest 2025-10-10T00:49:19.4655540Z inflating: build/bin/HashStoreTest 2025-10-10T00:49:19.4710079Z inflating: build/bin/BackoffTest 2025-10-10T00:49:19.4765076Z inflating: build/bin/FileStoreTest 2025-10-10T00:49:19.5839433Z inflating: build/bin/test_jit 2025-10-10T00:49:19.5900704Z inflating: build/bin/test_dist_autograd 2025-10-10T00:49:19.5903309Z inflating: build/bin/example_allreduce 2025-10-10T00:49:19.5973644Z inflating: build/bin/ProcessGroupGlooTest 2025-10-10T00:49:19.6041814Z inflating: build/bin/test_cpp_rpc 2025-10-10T00:49:19.6045146Z inflating: build/bin/parallel_benchmark 2025-10-10T00:49:19.7153513Z inflating: build/bin/test_api 2025-10-10T00:49:19.7484866Z inflating: build/bin/test_lazy 2025-10-10T00:49:19.7489379Z inflating: build/bin/torch_shm_manager 2025-10-10T00:49:19.7490185Z creating: .additional_ci_files/ 2025-10-10T00:49:19.7552417Z inflating: .additional_ci_files/test-times.json 2025-10-10T00:49:19.7794042Z inflating: .additional_ci_files/test-class-times.json 2025-10-10T00:49:19.7814605Z ##[group]Run rm artifacts.zip 2025-10-10T00:49:19.7814838Z rm artifacts.zip 2025-10-10T00:49:19.7819525Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:19.7819786Z env: 2025-10-10T00:49:19.7819944Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:19.7820130Z ##[endgroup] 2025-10-10T00:49:19.8444225Z ##[group]Run df -H 2025-10-10T00:49:19.8444428Z df -H 2025-10-10T00:49:19.8448976Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:19.8449214Z env: 2025-10-10T00:49:19.8449375Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:19.8449552Z ##[endgroup] 2025-10-10T00:49:19.8492498Z Filesystem Size Used Avail Use% Mounted on 2025-10-10T00:49:19.8492823Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-10-10T00:49:19.8493439Z tmpfs 67G 0 67G 0% /dev/shm 2025-10-10T00:49:19.8493745Z tmpfs 27G 791k 27G 1% /run 2025-10-10T00:49:19.8494001Z /dev/nvme0n1p1 215G 71G 145G 33% / 2025-10-10T00:49:19.8494245Z tmpfs 67G 13k 67G 1% /tmp 2025-10-10T00:49:19.8494500Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-10-10T00:49:19.8519941Z Prepare all required actions 2025-10-10T00:49:19.8520896Z Getting action download info 2025-10-10T00:49:20.0099861Z ##[group]Run ./.github/actions/download-td-artifacts 2025-10-10T00:49:20.0100143Z with: 2025-10-10T00:49:20.0100312Z env: 2025-10-10T00:49:20.0100497Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.0100692Z ##[endgroup] 2025-10-10T00:49:20.0211048Z ##[group]Run seemethere/download-artifact-s3@v4 2025-10-10T00:49:20.0211298Z with: 2025-10-10T00:49:20.0211462Z name: td_results 2025-10-10T00:49:20.0211654Z s3-bucket: gha-artifacts 2025-10-10T00:49:20.0211859Z region: us-east-1 2025-10-10T00:49:20.0212057Z env: 2025-10-10T00:49:20.0212252Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.0212444Z ##[endgroup] 2025-10-10T00:49:20.6366818Z (node:48441) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-10-10T00:49:20.6372636Z 2025-10-10T00:49:20.6377812Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-10-10T00:49:20.6379432Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-10-10T00:49:20.6379907Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-10-10T00:49:20.7263930Z Found 0 objects with prefix pytorch/pytorch/18392306145/td_results/ 2025-10-10T00:49:20.7271615Z Artifact download has finished successfully 2025-10-10T00:49:20.7458067Z ##[group]Run mkdir -p .additional_ci_files 2025-10-10T00:49:20.7458335Z mkdir -p .additional_ci_files 2025-10-10T00:49:20.7458624Z mv td_results.json .additional_ci_files/td_results.json || true 2025-10-10T00:49:20.7464063Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:20.7464326Z env: 2025-10-10T00:49:20.7464502Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.7464702Z ##[endgroup] 2025-10-10T00:49:20.7516789Z mv: cannot stat 'td_results.json': No such file or directory 2025-10-10T00:49:20.7605898Z ##[group]Run .github/scripts/parse_ref.py 2025-10-10T00:49:20.7606179Z .github/scripts/parse_ref.py 2025-10-10T00:49:20.7611223Z shell: /usr/bin/bash -e {0} 2025-10-10T00:49:20.7611428Z env: 2025-10-10T00:49:20.7611601Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.7611792Z ##[endgroup] 2025-10-10T00:49:20.7916925Z Setting output branch=main 2025-10-10T00:49:20.8010633Z Prepare all required actions 2025-10-10T00:49:20.8011002Z Getting action download info 2025-10-10T00:49:20.9265445Z ##[group]Run ./.github/actions/filter-test-configs 2025-10-10T00:49:20.9265730Z with: 2025-10-10T00:49:20.9266408Z github-token: *** 2025-10-10T00:49:20.9268207Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-10-10T00:49:20.9270229Z job-name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:20.9270578Z env: 2025-10-10T00:49:20.9270757Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.9270960Z ##[endgroup] 2025-10-10T00:49:20.9309927Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T00:49:20.9310198Z with: 2025-10-10T00:49:20.9310382Z shell: bash 2025-10-10T00:49:20.9310583Z timeout_minutes: 10 2025-10-10T00:49:20.9310794Z max_attempts: 5 2025-10-10T00:49:20.9311005Z retry_wait_seconds: 30 2025-10-10T00:49:20.9311604Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-10-10T00:49:20.9312219Z polling_interval_seconds: 1 2025-10-10T00:49:20.9312478Z warning_on_retry: true 2025-10-10T00:49:20.9312700Z continue_on_error: false 2025-10-10T00:49:20.9312921Z env: 2025-10-10T00:49:20.9313107Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:20.9313516Z GITHUB_TOKEN: *** 2025-10-10T00:49:20.9313732Z ##[endgroup] 2025-10-10T00:49:21.0505921Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-10-10T00:49:21.2384214Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T00:49:22.0611278Z Collecting requests==2.27.1 2025-10-10T00:49:22.0764619Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-10-10T00:49:22.3186246Z Collecting pyyaml==6.0.2 2025-10-10T00:49:22.3251683Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-10-10T00:49:22.6798237Z Collecting charset-normalizer~=2.0.0 2025-10-10T00:49:22.6837012Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-10-10T00:49:22.7294363Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-10-10T00:49:22.7301249Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-10-10T00:49:22.8328029Z Collecting certifi>=2017.4.17 2025-10-10T00:49:22.8388572Z Downloading certifi-2025.10.5-py3-none-any.whl (163 kB) 2025-10-10T00:49:22.9450518Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-10-10T00:49:23.4471402Z Successfully installed certifi-2025.10.5 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-10-10T00:49:23.9972226Z Command completed after 1 attempt(s). 2025-10-10T00:49:24.0211645Z ##[group]Run set -x 2025-10-10T00:49:24.0211873Z set -x 2025-10-10T00:49:24.0212046Z  2025-10-10T00:49:24.0212316Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-10-10T00:49:24.0212646Z # in runner workspace 2025-10-10T00:49:24.0213079Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-10-10T00:49:24.0218232Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:24.0218495Z env: 2025-10-10T00:49:24.0218661Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.0218862Z ##[endgroup] 2025-10-10T00:49:24.0244473Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-10-10T00:49:24.0387697Z Setting output branch=main 2025-10-10T00:49:24.0516029Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-10-10T00:49:24.0516359Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-10-10T00:49:24.0516623Z echo "Job name: ${JOB_NAME}" 2025-10-10T00:49:24.0516843Z  2025-10-10T00:49:24.0517123Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-10-10T00:49:24.0517460Z # in runner workspace 2025-10-10T00:49:24.0517789Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-10-10T00:49:24.0518140Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-10-10T00:49:24.0518396Z  --job-name "${JOB_NAME}" \ 2025-10-10T00:49:24.0520202Z  --test-matrix "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" \ 2025-10-10T00:49:24.0522074Z  --selected-test-configs "" \ 2025-10-10T00:49:24.0522337Z  --pr-number "${PR_NUMBER}" \ 2025-10-10T00:49:24.0522574Z  --tag "${TAG}" \ 2025-10-10T00:49:24.0522791Z  --event-name "${EVENT_NAME}" \ 2025-10-10T00:49:24.0523034Z  --schedule "${SCHEDULE}" \ 2025-10-10T00:49:24.0523300Z  --branch "${HEAD_BRANCH}" 2025-10-10T00:49:24.0528297Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:24.0528563Z env: 2025-10-10T00:49:24.0528737Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.0529316Z GITHUB_TOKEN: *** 2025-10-10T00:49:24.0529673Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:24.0530023Z PR_NUMBER: 2025-10-10T00:49:24.0530191Z TAG: 2025-10-10T00:49:24.0530358Z EVENT_NAME: push 2025-10-10T00:49:24.0530544Z SCHEDULE: 2025-10-10T00:49:24.0530719Z HEAD_BRANCH: main 2025-10-10T00:49:24.0530902Z ##[endgroup] 2025-10-10T00:49:24.0557561Z Workflow: inductor 2025-10-10T00:49:24.0560027Z Job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:24.2484321Z Setting output keep-going=True 2025-10-10T00:49:24.2484864Z Setting output ci-verbose-test-logs=False 2025-10-10T00:49:24.2485260Z Setting output ci-test-showlocals=False 2025-10-10T00:49:24.2485568Z Setting output ci-no-test-timeout=False 2025-10-10T00:49:24.2486296Z Setting output ci-no-td=False 2025-10-10T00:49:24.2486599Z Setting output ci-td-distributed=False 2025-10-10T00:49:24.2486841Z Setting output is-unstable=False 2025-10-10T00:49:24.2487059Z Setting output reenabled-issues= 2025-10-10T00:49:24.2488987Z Setting output test-matrix={"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-10-10T00:49:24.2490812Z Setting output is-test-matrix-empty=False 2025-10-10T00:49:24.2758693Z ##[group]Run echo "Filtered matrix:" 2025-10-10T00:49:24.2758956Z echo "Filtered matrix:" 2025-10-10T00:49:24.2760529Z echo "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" 2025-10-10T00:49:24.2762210Z  2025-10-10T00:49:24.2762378Z echo 2025-10-10T00:49:24.2762581Z echo "Is the current job unstable? False" 2025-10-10T00:49:24.2762825Z  2025-10-10T00:49:24.2762983Z echo 2025-10-10T00:49:24.2763178Z echo "Is keep-going label set? True" 2025-10-10T00:49:24.2763400Z  2025-10-10T00:49:24.2763558Z echo 2025-10-10T00:49:24.2763738Z echo "Reenabled issues? " 2025-10-10T00:49:24.2768685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:24.2768946Z env: 2025-10-10T00:49:24.2769133Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.2769342Z ##[endgroup] 2025-10-10T00:49:24.2793452Z Filtered matrix: 2025-10-10T00:49:24.2795421Z {include: [{config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: linux.24xl.spr-metal}]} 2025-10-10T00:49:24.2797099Z 2025-10-10T00:49:24.2797196Z Is the current job unstable? False 2025-10-10T00:49:24.2797347Z 2025-10-10T00:49:24.2797448Z Is keep-going label set? True 2025-10-10T00:49:24.2797597Z 2025-10-10T00:49:24.2797680Z Reenabled issues? 2025-10-10T00:49:24.2912627Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-10-10T00:49:24.2912991Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-10-10T00:49:24.2917520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:24.2917767Z env: 2025-10-10T00:49:24.2917928Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.2918108Z JOB_TIMEOUT: 240 2025-10-10T00:49:24.2918270Z ##[endgroup] 2025-10-10T00:49:24.3004415Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:49:24.3004749Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:49:24.3005026Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-10-10T00:49:24.3009269Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T00:49:24.3009509Z env: 2025-10-10T00:49:24.3009694Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.3010002Z ##[endgroup] 2025-10-10T00:49:24.3189746Z ##[group]Run set -x 2025-10-10T00:49:24.3190015Z set -x 2025-10-10T00:49:24.3190182Z  2025-10-10T00:49:24.3190361Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-10-10T00:49:24.3190629Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-10-10T00:49:24.3190896Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-10-10T00:49:24.3191135Z  TEST_COMMAND=.ci/onnx/test.sh 2025-10-10T00:49:24.3191332Z else 2025-10-10T00:49:24.3191518Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-10-10T00:49:24.3191730Z fi 2025-10-10T00:49:24.3191876Z  2025-10-10T00:49:24.3192050Z # Leaving 1GB for the runner and other things 2025-10-10T00:49:24.3192406Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-10-10T00:49:24.3192960Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-10-10T00:49:24.3193408Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-10-10T00:49:24.3193731Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-10-10T00:49:24.3193992Z  2025-10-10T00:49:24.3194181Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-10-10T00:49:24.3194405Z  SHM_OPTS= 2025-10-10T00:49:24.3194575Z  JENKINS_USER= 2025-10-10T00:49:24.3194814Z  # ensure that docker container cleanly exits in 12 hours 2025-10-10T00:49:24.3195115Z  # if for some reason cleanup action doesn't stop container 2025-10-10T00:49:24.3195379Z  # when job is cancelled 2025-10-10T00:49:24.3195590Z  DOCKER_SHELL_CMD="sleep 12h" 2025-10-10T00:49:24.3195778Z else 2025-10-10T00:49:24.3195962Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-10-10T00:49:24.3196188Z  JENKINS_USER="--user jenkins" 2025-10-10T00:49:24.3196402Z  DOCKER_SHELL_CMD= 2025-10-10T00:49:24.3196576Z fi 2025-10-10T00:49:24.3196724Z  2025-10-10T00:49:24.3196951Z # detached container should get cleaned up by teardown_ec2_linux 2025-10-10T00:49:24.3197284Z # TODO: Stop building test binaries as part of the build phase 2025-10-10T00:49:24.3197648Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-10-10T00:49:24.3197980Z # shellcheck disable=SC2086,SC2090 2025-10-10T00:49:24.3198207Z container_name=$(docker run \ 2025-10-10T00:49:24.3198424Z  ${GPU_FLAG:-} \ 2025-10-10T00:49:24.3198635Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-10-10T00:49:24.3198857Z  -e BUILD_ENVIRONMENT \ 2025-10-10T00:49:24.3199062Z  -e PR_NUMBER \ 2025-10-10T00:49:24.3199255Z  -e GITHUB_ACTIONS \ 2025-10-10T00:49:24.3199454Z  -e GITHUB_REPOSITORY \ 2025-10-10T00:49:24.3199656Z  -e GITHUB_WORKFLOW \ 2025-10-10T00:49:24.3199853Z  -e GITHUB_JOB \ 2025-10-10T00:49:24.3200037Z  -e GITHUB_RUN_ID \ 2025-10-10T00:49:24.3200228Z  -e GITHUB_RUN_NUMBER \ 2025-10-10T00:49:24.3200422Z  -e GITHUB_RUN_ATTEMPT \ 2025-10-10T00:49:24.3200619Z  -e JOB_ID \ 2025-10-10T00:49:24.3200804Z  -e JOB_NAME \ 2025-10-10T00:49:24.3200995Z  -e BASE_SHA \ 2025-10-10T00:49:24.3201174Z  -e BRANCH \ 2025-10-10T00:49:24.3201352Z  -e SHA1 \ 2025-10-10T00:49:24.3201540Z  -e AWS_DEFAULT_REGION \ 2025-10-10T00:49:24.3201752Z  -e IN_WHEEL_TEST \ 2025-10-10T00:49:24.3201944Z  -e SHARD_NUMBER \ 2025-10-10T00:49:24.3202145Z  -e TEST_CONFIG \ 2025-10-10T00:49:24.3202344Z  -e NUM_TEST_SHARDS \ 2025-10-10T00:49:24.3202555Z  -e REENABLED_ISSUES \ 2025-10-10T00:49:24.3202768Z  -e CONTINUE_THROUGH_ERROR \ 2025-10-10T00:49:24.3203187Z  -e VERBOSE_TEST_LOGS \ 2025-10-10T00:49:24.3203402Z  -e TEST_SHOWLOCALS \ 2025-10-10T00:49:24.3203591Z  -e NO_TEST_TIMEOUT \ 2025-10-10T00:49:24.3203765Z  -e NO_TD \ 2025-10-10T00:49:24.3203937Z  -e TD_DISTRIBUTED \ 2025-10-10T00:49:24.3204126Z  -e PR_LABELS \ 2025-10-10T00:49:24.3204326Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-10-10T00:49:24.3204539Z  -e SCCACHE_BUCKET \ 2025-10-10T00:49:24.3204730Z  -e SCCACHE_REGION \ 2025-10-10T00:49:24.3204914Z  -e XLA_CUDA \ 2025-10-10T00:49:24.3205110Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-10-10T00:49:24.3205342Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-10-10T00:49:24.3205572Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-10-10T00:49:24.3205808Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-10-10T00:49:24.3206029Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-10-10T00:49:24.3206253Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-10-10T00:49:24.3206468Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-10-10T00:49:24.3206676Z  -e DASHBOARD_TAG \ 2025-10-10T00:49:24.3206868Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-10-10T00:49:24.3207103Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-10-10T00:49:24.3207361Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-10-10T00:49:24.3207626Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-10-10T00:49:24.3207879Z  --security-opt seccomp=unconfined \ 2025-10-10T00:49:24.3208101Z  --cap-add=SYS_PTRACE \ 2025-10-10T00:49:24.3208298Z  --ipc=host \ 2025-10-10T00:49:24.3208468Z  ${SHM_OPTS} \ 2025-10-10T00:49:24.3208641Z  --tty \ 2025-10-10T00:49:24.3208806Z  --detach \ 2025-10-10T00:49:24.3208988Z  --name="${container_name}" \ 2025-10-10T00:49:24.3209186Z  ${JENKINS_USER} \ 2025-10-10T00:49:24.3209420Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-10-10T00:49:24.3209669Z  -w /var/lib/jenkins/workspace \ 2025-10-10T00:49:24.3209876Z  "${DOCKER_IMAGE}" \ 2025-10-10T00:49:24.3210055Z  ${DOCKER_SHELL_CMD} 2025-10-10T00:49:24.3210232Z ) 2025-10-10T00:49:24.3210455Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-10-10T00:49:24.3210706Z  2025-10-10T00:49:24.3210882Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-10-10T00:49:24.3211289Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-10-10T00:49:24.3211628Z fi 2025-10-10T00:49:24.3211791Z  2025-10-10T00:49:24.3212120Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-10-10T00:49:24.3216875Z shell: /usr/bin/bash -e {0} 2025-10-10T00:49:24.3217087Z env: 2025-10-10T00:49:24.3217268Z GIT_DEFAULT_BRANCH: main 2025-10-10T00:49:24.3217514Z BUILD_ENVIRONMENT: linux-jammy-py3.10-gcc11-build 2025-10-10T00:49:24.3217758Z PR_NUMBER: 2025-10-10T00:49:24.3217965Z GITHUB_REPOSITORY: pytorch/pytorch 2025-10-10T00:49:24.3218226Z GITHUB_WORKFLOW: inductor 2025-10-10T00:49:24.3218416Z GITHUB_JOB: test 2025-10-10T00:49:24.3218587Z GITHUB_RUN_ID: 18392306145 2025-10-10T00:49:24.3218783Z GITHUB_RUN_NUMBER: 158985 2025-10-10T00:49:24.3218974Z GITHUB_RUN_ATTEMPT: 1 2025-10-10T00:49:24.3219159Z JOB_ID: 52406535390 2025-10-10T00:49:24.3219470Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:24.3219801Z BRANCH: main 2025-10-10T00:49:24.3219999Z SHA1: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:24.3220261Z BASE_SHA: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:24.3220520Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-10-10T00:49:24.3220752Z SHARD_NUMBER: 1 2025-10-10T00:49:24.3221065Z NUM_TEST_SHARDS: 1 2025-10-10T00:49:24.3221248Z EXTRA_FLAGS: 2025-10-10T00:49:24.3221414Z OP_BENCHMARK_TESTS: 2025-10-10T00:49:24.3221598Z REENABLED_ISSUES: 2025-10-10T00:49:24.3221788Z CONTINUE_THROUGH_ERROR: True 2025-10-10T00:49:24.3221993Z VERBOSE_TEST_LOGS: False 2025-10-10T00:49:24.3222184Z TEST_SHOWLOCALS: False 2025-10-10T00:49:24.3222378Z NO_TEST_TIMEOUT: False 2025-10-10T00:49:24.3222562Z NO_TD: False 2025-10-10T00:49:24.3222733Z TD_DISTRIBUTED: False 2025-10-10T00:49:24.3222954Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-10-10T00:49:24.3223212Z SCCACHE_REGION: us-east-1 2025-10-10T00:49:24.3223403Z SHM_SIZE: 1g 2025-10-10T00:49:24.3223933Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:49:24.3224481Z XLA_CUDA: 2025-10-10T00:49:24.3224734Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-10-10T00:49:24.3225052Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-10-10T00:49:24.3225283Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-10-10T00:49:24.3225496Z DASHBOARD_TAG: 2025-10-10T00:49:24.3225864Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-10-10T00:49:24.3226266Z HUGGING_FACE_HUB_TOKEN: *** 2025-10-10T00:49:24.3226568Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-10-10T00:49:24.3226945Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T00:49:24.3227324Z ##[endgroup] 2025-10-10T00:49:24.3251614Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-10-10T00:49:24.3251970Z + [[ linux-jammy-py3.10-gcc11-build == *onnx* ]] 2025-10-10T00:49:24.3252242Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-10-10T00:49:24.3252561Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-10-10T00:49:24.3898834Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-10-10T00:49:24.3899174Z + TOTAL_MEMORY_WITH_SWAP=125 2025-10-10T00:49:24.3899477Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-10-10T00:49:24.3899739Z + SHM_OPTS=--shm-size=1g 2025-10-10T00:49:24.3899974Z + JENKINS_USER='--user jenkins' 2025-10-10T00:49:24.3900184Z + DOCKER_SHELL_CMD= 2025-10-10T00:49:24.3905683Z +++ nproc --ignore=2 2025-10-10T00:49:24.4098780Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_18392306145 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T00:49:34.2913981Z + container_name=3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T00:49:34.2914670Z + echo DOCKER_CONTAINER_ID=3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T00:49:34.2915127Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-10-10T00:49:34.2919735Z ++ echo dist/torch-2.10.0a0+git344e636-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:34.2922436Z + docker exec -t 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c sh -c 'python3 -m pip install dist/torch-2.10.0a0+git344e636-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-10-10T00:49:34.7150892Z Processing ./dist/torch-2.10.0a0+git344e636-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+git344e636) 2025-10-10T00:49:34.9327193Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.18.0) 2025-10-10T00:49:34.9328158Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (4.15.0) 2025-10-10T00:49:34.9333619Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.13.3) 2025-10-10T00:49:34.9334430Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (2.8.8) 2025-10-10T00:49:34.9340095Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.1.6) 2025-10-10T00:49:34.9340861Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (2025.9.0) 2025-10-10T00:49:34.9354670Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.3.0) 2025-10-10T00:49:34.9650450Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.22.4) 2025-10-10T00:49:34.9662576Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (1.3.0) 2025-10-10T00:49:34.9705710Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+git344e636->torch==2.10.0a0+git344e636) (3.0.3) 2025-10-10T00:49:35.7406205Z Installing collected packages: torch 2025-10-10T00:49:43.2208412Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-10-10T00:49:43.2209135Z dall-e 0.1 requires torchvision, which is not installed. 2025-10-10T00:49:43.2209468Z effdet 0.4.1 requires torchvision, which is not installed. 2025-10-10T00:49:43.2209816Z python-doctr 1.0.0 requires torchvision>=0.15.0, which is not installed. 2025-10-10T00:49:43.2210279Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-10-10T00:49:43.2210852Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-10-10T00:49:43.2211388Z timm 1.0.14 requires torchvision, which is not installed. 2025-10-10T00:49:43.2211776Z Successfully installed torch-2.10.0a0+git344e636 2025-10-10T00:49:43.3419940Z + export TERM=vt100 2025-10-10T00:49:43.3420815Z + TERM=vt100 2025-10-10T00:49:43.3421147Z ++ dirname .ci/pytorch/test.sh 2025-10-10T00:49:43.3428590Z + source .ci/pytorch/common.sh 2025-10-10T00:49:43.3432130Z +++ dirname .ci/pytorch/common.sh 2025-10-10T00:49:43.3439072Z ++ source .ci/pytorch/common_utils.sh 2025-10-10T00:49:43.3439379Z +++ declare -f -t trap_add 2025-10-10T00:49:43.3443631Z ++ set -ex -o pipefail 2025-10-10T00:49:43.3444205Z ++ [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-10-10T00:49:43.3444506Z ++ BUILD_TEST_LIBTORCH=0 2025-10-10T00:49:43.3449042Z ++ dirname .ci/pytorch/test.sh 2025-10-10T00:49:43.3454540Z + source .ci/pytorch/common-build.sh 2025-10-10T00:49:43.3461354Z ++ [[ linux-jammy-py3.10-gcc11-build != *win-* ]] 2025-10-10T00:49:43.3464789Z ++++ dirname .ci/pytorch/common-build.sh 2025-10-10T00:49:43.3477397Z +++ cd .ci/pytorch 2025-10-10T00:49:43.3477642Z +++ pwd -P 2025-10-10T00:49:43.3479604Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-10-10T00:49:43.3480493Z ++ [[ linux-jammy-py3.10-gcc11-build == *-pch* ]] 2025-10-10T00:49:43.3483225Z ++ which sccache 2025-10-10T00:49:43.3501013Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-10-10T00:49:43.3501348Z ++ sccache --stop-server 2025-10-10T00:49:43.3523271Z ++ true 2025-10-10T00:49:43.3523732Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-10-10T00:49:43.3537457Z ++ trap_add sccache_epilogue EXIT 2025-10-10T00:49:43.3537723Z ++ trap_add_cmd=sccache_epilogue 2025-10-10T00:49:43.3537932Z ++ shift 2025-10-10T00:49:43.3538120Z ++ for trap_add_name in "$@" 2025-10-10T00:49:43.3542955Z ++++ trap -p EXIT 2025-10-10T00:49:43.3543223Z +++ eval 'extract_trap_cmd ' 2025-10-10T00:49:43.3543435Z ++++ extract_trap_cmd 2025-10-10T00:49:43.3543677Z ++++ printf '%s\n' '' 2025-10-10T00:49:43.3544033Z +++ printf '%s\n' sccache_epilogue 2025-10-10T00:49:43.3544294Z ++ trap -- ' 2025-10-10T00:49:43.3544479Z sccache_epilogue' EXIT 2025-10-10T00:49:43.3544673Z ++ [[ -n 1 ]] 2025-10-10T00:49:43.3544976Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-10-10T00:49:43.3545402Z Skipping sccache server initialization, setting environment variables 2025-10-10T00:49:43.3545731Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-10-10T00:49:43.3546207Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-10-10T00:49:43.3546490Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T00:49:43.3546823Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T00:49:43.3554152Z ++ export RUST_LOG=sccache::server=error 2025-10-10T00:49:43.3554452Z ++ RUST_LOG=sccache::server=error 2025-10-10T00:49:43.3554681Z ++ sccache --zero-stats 2025-10-10T00:49:43.5018487Z Statistics zeroed. 2025-10-10T00:49:43.5021839Z ++ which ccache 2025-10-10T00:49:43.5058766Z + [[ linux-jammy-py3.10-gcc11-build != *rocm* ]] 2025-10-10T00:49:43.5059128Z + [[ linux-jammy-py3.10-gcc11-build != *s390x* ]] 2025-10-10T00:49:43.5059437Z + [[ -d /var/lib/jenkins/workspace ]] 2025-10-10T00:49:43.5059705Z ++ stat -c %u /var/lib/jenkins/workspace 2025-10-10T00:49:43.5076436Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-10-10T00:49:43.5076759Z + trap_add cleanup_workspace EXIT 2025-10-10T00:49:43.5076996Z + trap_add_cmd=cleanup_workspace 2025-10-10T00:49:43.5077205Z + shift 2025-10-10T00:49:43.5077383Z + for trap_add_name in "$@" 2025-10-10T00:49:43.5077604Z +++ trap -p EXIT 2025-10-10T00:49:43.5077798Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-10-10T00:49:43.5078033Z sccache_epilogue'\'' EXIT' 2025-10-10T00:49:43.5078246Z +++ extract_trap_cmd trap -- ' 2025-10-10T00:49:43.5078450Z sccache_epilogue' EXIT 2025-10-10T00:49:43.5078630Z +++ printf '%s\n' ' 2025-10-10T00:49:43.5078810Z sccache_epilogue' 2025-10-10T00:49:43.5079001Z ++ printf '%s\n' cleanup_workspace 2025-10-10T00:49:43.5079285Z + trap -- ' 2025-10-10T00:49:43.5079467Z sccache_epilogue 2025-10-10T00:49:43.5079790Z cleanup_workspace' EXIT 2025-10-10T00:49:43.5080613Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-10-10T00:49:43.9573498Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-10-10T00:49:43.9585224Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-10-10T00:49:43.9585515Z + echo 'Environment variables:' 2025-10-10T00:49:43.9586333Z Environment variables: 2025-10-10T00:49:43.9586669Z + env 2025-10-10T00:49:43.9599741Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T00:49:43.9600228Z CONTINUE_THROUGH_ERROR=True 2025-10-10T00:49:43.9600577Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-10-10T00:49:43.9601243Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-10-10T00:49:43.9601953Z HOSTNAME=3283ba949f9d 2025-10-10T00:49:43.9602950Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9603858Z GITHUB_ACTION=__run_2 2025-10-10T00:49:43.9604069Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-10-10T00:49:43.9604283Z GITHUB_RUN_NUMBER=158985 2025-10-10T00:49:43.9604505Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-10-10T00:49:43.9604744Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-10-10T00:49:43.9604976Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-10-10T00:49:43.9605190Z SCCACHE_IDLE_TIMEOUT=0 2025-10-10T00:49:43.9605666Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-10-10T00:49:43.9605882Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-10-10T00:49:43.9606099Z GITHUB_REF_TYPE=branch 2025-10-10T00:49:43.9606300Z BASE_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9606636Z XLA_CUDA= 2025-10-10T00:49:43.9606831Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-10-10T00:49:43.9607112Z HUGGING_FACE_HUB_TOKEN=*** 2025-10-10T00:49:43.9607562Z *** 2025-10-10T00:49:43.9607729Z GITHUB_REPOSITORY_ID=65600975 2025-10-10T00:49:43.9607927Z GITHUB_ACTIONS=true 2025-10-10T00:49:43.9608140Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T00:49:43.9608405Z SHA1=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9608654Z GITHUB_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9608994Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-10-10T00:49:43.9609291Z UCC_HOME=/usr 2025-10-10T00:49:43.9609462Z VERBOSE_TEST_LOGS=False 2025-10-10T00:49:43.9609645Z GITHUB_REF=refs/heads/main 2025-10-10T00:49:43.9609828Z SHARD_NUMBER=1 2025-10-10T00:49:43.9609992Z GITHUB_REF_PROTECTED=true 2025-10-10T00:49:43.9610176Z HOME=/var/lib/jenkins 2025-10-10T00:49:43.9610376Z GITHUB_API_URL=https://api.github.com 2025-10-10T00:49:43.9610604Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-10-10T00:49:43.9610793Z UCX_COMMIT= 2025-10-10T00:49:43.9610946Z USE_SYSTEM_NCCL=1 2025-10-10T00:49:43.9611114Z NUM_TEST_SHARDS=1 2025-10-10T00:49:43.9611280Z UCX_HOME=/usr 2025-10-10T00:49:43.9611676Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9612230Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:43.9612752Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9613265Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-10-10T00:49:43.9613590Z GITHUB_EVENT_NAME=push 2025-10-10T00:49:43.9613767Z DASHBOARD_TAG= 2025-10-10T00:49:43.9613943Z GITHUB_RUN_ID=18392306145 2025-10-10T00:49:43.9614134Z INSTALLED_OPENBLAS= 2025-10-10T00:49:43.9614544Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9614977Z GITHUB_ACTOR=pytorchmergebot 2025-10-10T00:49:43.9615179Z PR_NUMBER= 2025-10-10T00:49:43.9615341Z DESIRED_CUDA= 2025-10-10T00:49:43.9615510Z GITHUB_RUN_ATTEMPT=1 2025-10-10T00:49:43.9615707Z ANACONDA_PYTHON_VERSION=3.10 2025-10-10T00:49:43.9615959Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-10-10T00:49:43.9616208Z TERM=vt100 2025-10-10T00:49:43.9616377Z INSTALLED_VISION=yes 2025-10-10T00:49:43.9616553Z BRANCH=main 2025-10-10T00:49:43.9616728Z SCCACHE_REGION=us-east-1 2025-10-10T00:49:43.9616939Z OPENSSL_ROOT_DIR=/opt/openssl 2025-10-10T00:49:43.9617151Z CUDA_PATH=/usr/local/cuda 2025-10-10T00:49:43.9617505Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-10-10T00:49:43.9617910Z GITHUB_SERVER_URL=https://github.com 2025-10-10T00:49:43.9618135Z UCC_COMMIT= 2025-10-10T00:49:43.9618301Z REENABLED_ISSUES= 2025-10-10T00:49:43.9618468Z DOCS=yes 2025-10-10T00:49:43.9618625Z SHLVL=1 2025-10-10T00:49:43.9618784Z MAX_JOBS=30 2025-10-10T00:49:43.9618944Z GITHUB_ACTOR_ID=97764156 2025-10-10T00:49:43.9619195Z GITHUB_WORKFLOW_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9619465Z GITHUB_REF_NAME=main 2025-10-10T00:49:43.9619869Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-10-10T00:49:43.9620172Z GITHUB_JOB=test 2025-10-10T00:49:43.9620392Z NO_TEST_TIMEOUT=False 2025-10-10T00:49:43.9620595Z TD_DISTRIBUTED=False 2025-10-10T00:49:43.9620797Z GITHUB_REPOSITORY=pytorch/pytorch 2025-10-10T00:49:43.9621016Z GITHUB_RETENTION_DAYS=90 2025-10-10T00:49:43.9621219Z OPENSSL_DIR=/opt/openssl 2025-10-10T00:49:43.9621422Z GITHUB_ACTION_REPOSITORY= 2025-10-10T00:49:43.9621949Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T00:49:43.9622459Z GITHUB_BASE_REF= 2025-10-10T00:49:43.9622631Z INSTALLED_ACL= 2025-10-10T00:49:43.9622958Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T00:49:43.9623325Z CI=true 2025-10-10T00:49:43.9623501Z GITHUB_REPOSITORY_OWNER=pytorch 2025-10-10T00:49:43.9623761Z RUST_LOG=sccache::server=error 2025-10-10T00:49:43.9623977Z JOB_ID=52406535390 2025-10-10T00:49:43.9624155Z GITHUB_HEAD_REF= 2025-10-10T00:49:43.9624333Z GITHUB_ACTION_REF= 2025-10-10T00:49:43.9624548Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-10-10T00:49:43.9624801Z TEST_SHOWLOCALS=False 2025-10-10T00:49:43.9624997Z GITHUB_WORKFLOW=inductor 2025-10-10T00:49:43.9625201Z DEBIAN_FRONTEND=noninteractive 2025-10-10T00:49:43.9625613Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9626228Z NO_TD=False 2025-10-10T00:49:43.9626417Z SKIP_SCCACHE_INITIALIZATION=1 2025-10-10T00:49:43.9626648Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-10-10T00:49:43.9626871Z _=/usr/bin/env 2025-10-10T00:49:43.9627120Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-10-10T00:49:43.9848555Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-10-10T00:49:43.9849072Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-10-10T00:49:43.9849473Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-10-10T00:49:43.9849867Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-10-10T00:49:43.9850155Z + BUILD_DIR=build 2025-10-10T00:49:43.9850346Z + BUILD_RENAMED_DIR=build_renamed 2025-10-10T00:49:43.9850553Z + BUILD_BIN_DIR=build/bin 2025-10-10T00:49:43.9850728Z + SHARD_NUMBER=1 2025-10-10T00:49:43.9850978Z + NUM_TEST_SHARDS=1 2025-10-10T00:49:43.9851183Z + export TORCH_SERIALIZATION_DEBUG=1 2025-10-10T00:49:43.9851398Z + TORCH_SERIALIZATION_DEBUG=1 2025-10-10T00:49:43.9851588Z + export VALGRIND=ON 2025-10-10T00:49:43.9851767Z + VALGRIND=ON 2025-10-10T00:49:43.9851974Z + [[ linux-jammy-py3.10-gcc11-build == *clang9* ]] 2025-10-10T00:49:43.9852279Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-10-10T00:49:43.9852499Z + detect_cuda_arch 2025-10-10T00:49:43.9852700Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-10-10T00:49:43.9852959Z + [[ linux-jammy-py3.10-gcc11-build == *s390x* ]] 2025-10-10T00:49:43.9853193Z + [[ 0 == \1 ]] 2025-10-10T00:49:43.9853345Z + [[ True == \1 ]] 2025-10-10T00:49:43.9853540Z + [[ linux-jammy-py3.10-gcc11-build != *bazel* ]] 2025-10-10T00:49:43.9857641Z ++ realpath build/custom_test_artifacts 2025-10-10T00:49:43.9865675Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-10-10T00:49:43.9866267Z + [[ -n '' ]] 2025-10-10T00:49:43.9866484Z + echo 'Environment variables' 2025-10-10T00:49:43.9866719Z Environment variables 2025-10-10T00:49:43.9866914Z + env 2025-10-10T00:49:43.9885794Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T00:49:43.9886245Z CONTINUE_THROUGH_ERROR=True 2025-10-10T00:49:43.9886507Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-10-10T00:49:43.9887044Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-10-10T00:49:43.9887267Z HOSTNAME=3283ba949f9d 2025-10-10T00:49:43.9888028Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9888580Z GITHUB_ACTION=__run_2 2025-10-10T00:49:43.9888781Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-10-10T00:49:43.9889010Z GITHUB_RUN_NUMBER=158985 2025-10-10T00:49:43.9889243Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-10-10T00:49:43.9889497Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-10-10T00:49:43.9889740Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-10-10T00:49:43.9889971Z SCCACHE_IDLE_TIMEOUT=0 2025-10-10T00:49:43.9890263Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-10-10T00:49:43.9890496Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-10-10T00:49:43.9890718Z GITHUB_REF_TYPE=branch 2025-10-10T00:49:43.9890943Z BASE_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9891183Z XLA_CUDA= 2025-10-10T00:49:43.9891367Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-10-10T00:49:43.9891863Z HUGGING_FACE_HUB_TOKEN=*** 2025-10-10T00:49:43.9892137Z *** 2025-10-10T00:49:43.9892320Z GITHUB_REPOSITORY_ID=65600975 2025-10-10T00:49:43.9892538Z GITHUB_ACTIONS=true 2025-10-10T00:49:43.9892763Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-10-10T00:49:43.9893034Z SHA1=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9893301Z GITHUB_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9893659Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-10-10T00:49:43.9893986Z UCC_HOME=/usr 2025-10-10T00:49:43.9894161Z TORCH_SERIALIZATION_DEBUG=1 2025-10-10T00:49:43.9894370Z VERBOSE_TEST_LOGS=False 2025-10-10T00:49:43.9894568Z GITHUB_REF=refs/heads/main 2025-10-10T00:49:43.9894811Z SHARD_NUMBER=1 2025-10-10T00:49:43.9894999Z GITHUB_REF_PROTECTED=true 2025-10-10T00:49:43.9895199Z HOME=/var/lib/jenkins 2025-10-10T00:49:43.9895415Z GITHUB_API_URL=https://api.github.com 2025-10-10T00:49:43.9895663Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-10-10T00:49:43.9895873Z UCX_COMMIT= 2025-10-10T00:49:43.9896040Z USE_SYSTEM_NCCL=1 2025-10-10T00:49:43.9896223Z NUM_TEST_SHARDS=1 2025-10-10T00:49:43.9896401Z UCX_HOME=/usr 2025-10-10T00:49:43.9896782Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9897346Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T00:49:43.9897889Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9898413Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-10-10T00:49:43.9898752Z GITHUB_EVENT_NAME=push 2025-10-10T00:49:43.9898939Z DASHBOARD_TAG= 2025-10-10T00:49:43.9899120Z GITHUB_RUN_ID=18392306145 2025-10-10T00:49:43.9899318Z INSTALLED_OPENBLAS= 2025-10-10T00:49:43.9899725Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9900162Z GITHUB_ACTOR=pytorchmergebot 2025-10-10T00:49:43.9900375Z PR_NUMBER= 2025-10-10T00:49:43.9900540Z DESIRED_CUDA= 2025-10-10T00:49:43.9900713Z GITHUB_RUN_ATTEMPT=1 2025-10-10T00:49:43.9900889Z VALGRIND=ON 2025-10-10T00:49:43.9901065Z ANACONDA_PYTHON_VERSION=3.10 2025-10-10T00:49:43.9901312Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-10-10T00:49:43.9901556Z TERM=vt100 2025-10-10T00:49:43.9901716Z INSTALLED_VISION=yes 2025-10-10T00:49:43.9901897Z BRANCH=main 2025-10-10T00:49:43.9902070Z SCCACHE_REGION=us-east-1 2025-10-10T00:49:43.9902280Z OPENSSL_ROOT_DIR=/opt/openssl 2025-10-10T00:49:43.9902483Z CUDA_PATH=/usr/local/cuda 2025-10-10T00:49:43.9902843Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-10-10T00:49:43.9903233Z GITHUB_SERVER_URL=https://github.com 2025-10-10T00:49:43.9903455Z UCC_COMMIT= 2025-10-10T00:49:43.9903612Z REENABLED_ISSUES= 2025-10-10T00:49:43.9903786Z DOCS=yes 2025-10-10T00:49:43.9903947Z SHLVL=1 2025-10-10T00:49:43.9904097Z MAX_JOBS=30 2025-10-10T00:49:43.9904411Z GITHUB_ACTOR_ID=97764156 2025-10-10T00:49:43.9904669Z GITHUB_WORKFLOW_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T00:49:43.9904949Z GITHUB_REF_NAME=main 2025-10-10T00:49:43.9905235Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-10-10T00:49:43.9905547Z GITHUB_JOB=test 2025-10-10T00:49:43.9905737Z NO_TEST_TIMEOUT=False 2025-10-10T00:49:43.9906133Z TD_DISTRIBUTED=False 2025-10-10T00:49:43.9906344Z GITHUB_REPOSITORY=pytorch/pytorch 2025-10-10T00:49:43.9906608Z GITHUB_RETENTION_DAYS=90 2025-10-10T00:49:43.9906812Z OPENSSL_DIR=/opt/openssl 2025-10-10T00:49:43.9907017Z GITHUB_ACTION_REPOSITORY= 2025-10-10T00:49:43.9907536Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T00:49:43.9908053Z GITHUB_BASE_REF= 2025-10-10T00:49:43.9908240Z INSTALLED_ACL= 2025-10-10T00:49:43.9908597Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T00:49:43.9908956Z CI=true 2025-10-10T00:49:43.9909122Z GITHUB_REPOSITORY_OWNER=pytorch 2025-10-10T00:49:43.9909397Z RUST_LOG=sccache::server=error 2025-10-10T00:49:43.9909601Z JOB_ID=52406535390 2025-10-10T00:49:43.9909779Z GITHUB_HEAD_REF= 2025-10-10T00:49:43.9909951Z GITHUB_ACTION_REF= 2025-10-10T00:49:43.9910174Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-10-10T00:49:43.9910429Z TEST_SHOWLOCALS=False 2025-10-10T00:49:43.9910621Z GITHUB_WORKFLOW=inductor 2025-10-10T00:49:43.9910816Z DEBIAN_FRONTEND=noninteractive 2025-10-10T00:49:43.9911231Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_86de951f-2259-4dad-bf89-b96a5ee329ab 2025-10-10T00:49:43.9911643Z NO_TD=False 2025-10-10T00:49:43.9911826Z SKIP_SCCACHE_INITIALIZATION=1 2025-10-10T00:49:43.9912048Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-10-10T00:49:43.9912273Z _=/usr/bin/env 2025-10-10T00:49:43.9912448Z + echo 'Testing pytorch' 2025-10-10T00:49:43.9912646Z Testing pytorch 2025-10-10T00:49:43.9912831Z + export LANG=C.UTF-8 2025-10-10T00:49:43.9913012Z + LANG=C.UTF-8 2025-10-10T00:49:43.9913179Z + PR_NUMBER= 2025-10-10T00:49:43.9913401Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-10-10T00:49:43.9913717Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-10-10T00:49:43.9914025Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-10-10T00:49:43.9914325Z + [[ linux-jammy-py3.10-gcc11-build == *slow-gradcheck* ]] 2025-10-10T00:49:43.9914620Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-10-10T00:49:43.9914880Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-10-10T00:49:43.9915146Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-10-10T00:49:43.9915418Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-10-10T00:49:43.9915694Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-10-10T00:49:43.9915953Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-10-10T00:49:43.9916223Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-10-10T00:49:43.9916478Z + pip_install ninja==1.10.2 2025-10-10T00:49:43.9916754Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-10-10T00:49:43.9917084Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-10-10T00:49:44.4091712Z Collecting ninja==1.10.2 2025-10-10T00:49:44.4265610Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-10-10T00:49:44.4408769Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-10-10T00:49:45.2299260Z Installing collected packages: ninja 2025-10-10T00:49:45.2299600Z Attempting uninstall: ninja 2025-10-10T00:49:45.2302505Z Found existing installation: ninja 1.11.1.4 2025-10-10T00:49:45.2320829Z Uninstalling ninja-1.11.1.4: 2025-10-10T00:49:45.2553134Z Successfully uninstalled ninja-1.11.1.4 2025-10-10T00:49:45.3667236Z Successfully installed ninja-1.10.2 2025-10-10T00:49:45.4787726Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T00:49:45.4788862Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-10-10T00:49:45.4789605Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-10-10T00:49:45.4789941Z + [[ linux-jammy-py3.10-gcc11-build == *asan* ]] 2025-10-10T00:49:45.4790227Z + [[ linux-jammy-py3.10-gcc11-build == *-debug* ]] 2025-10-10T00:49:45.4790506Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-10-10T00:49:45.4790893Z + echo 'We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass' 2025-10-10T00:49:45.4791361Z We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass 2025-10-10T00:49:45.4791687Z + cd test 2025-10-10T00:49:45.4791967Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-10-10T00:49:45.7955933Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:49:45.7956836Z import pynvml # type: ignore[import] 2025-10-10T00:49:46.7479717Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-10-10T00:49:46.7480188Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-10-10T00:49:46.7480561Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-10-10T00:49:46.7480872Z + DYNAMO_BENCHMARK_FLAGS=() 2025-10-10T00:49:46.7481138Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-10-10T00:49:46.7481445Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-10-10T00:49:46.7481776Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-10-10T00:49:46.7482053Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-10-10T00:49:46.7482373Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-10-10T00:49:46.7482673Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-10-10T00:49:46.7482944Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-10-10T00:49:46.7483216Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-10-10T00:49:46.7483472Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-10-10T00:49:46.7483796Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-10-10T00:49:46.7484107Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-10-10T00:49:46.7484349Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-10-10T00:49:46.7601712Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-10-10T00:49:46.7602074Z + [[ linux-jammy-py3.10-gcc11-build == *-bazel-* ]] 2025-10-10T00:49:46.7604843Z + cd test 2025-10-10T00:49:46.7605111Z + python -c 'import torch; print(torch.__config__.show())' 2025-10-10T00:49:47.0652111Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:49:47.0653450Z import pynvml # type: ignore[import] 2025-10-10T00:49:47.7203042Z PyTorch built with: 2025-10-10T00:49:47.7203476Z - GCC 11.4 2025-10-10T00:49:47.7203702Z - C++ Version: 201703 2025-10-10T00:49:47.7204200Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-10-10T00:49:47.7204704Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-10-10T00:49:47.7205044Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-10-10T00:49:47.7205298Z - LAPACK is enabled (usually provided by MKL) 2025-10-10T00:49:47.7205981Z - NNPACK is enabled 2025-10-10T00:49:47.7206196Z - CPU capability usage: AVX512 2025-10-10T00:49:47.7209173Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=344e6365a0068c2d2847fcec0c55dd53291d475e, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-10-10T00:49:47.7212432Z 2025-10-10T00:49:47.9828890Z + cd test 2025-10-10T00:49:47.9829235Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-10-10T00:49:48.2885190Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:49:48.2886098Z import pynvml # type: ignore[import] 2025-10-10T00:49:48.9543854Z ATen/Parallel: 2025-10-10T00:49:48.9547015Z at::get_num_threads() : 16 2025-10-10T00:49:48.9547387Z at::get_num_interop_threads() : 16 2025-10-10T00:49:48.9553548Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-10-10T00:49:48.9558075Z omp_get_max_threads() : 16 2025-10-10T00:49:48.9558537Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-10-10T00:49:48.9558950Z mkl_get_max_threads() : 16 2025-10-10T00:49:48.9559254Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-10-10T00:49:48.9559608Z std::thread::hardware_concurrency() : 32 2025-10-10T00:49:48.9559844Z Environment variables: 2025-10-10T00:49:48.9560055Z OMP_NUM_THREADS : [not set] 2025-10-10T00:49:48.9560267Z MKL_NUM_THREADS : [not set] 2025-10-10T00:49:48.9560483Z ATen parallel backend: OpenMP 2025-10-10T00:49:48.9560616Z 2025-10-10T00:49:49.2246750Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-10-10T00:49:49.2248463Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-10-10T00:49:49.2248760Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-10-10T00:49:49.2249093Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-10-10T00:49:49.2249377Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-10-10T00:49:49.2249644Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-10-10T00:49:49.2249942Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-10-10T00:49:49.2250262Z + [[ dynamic_cpu_inductor_huggingface == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-10-10T00:49:49.2250592Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-10-10T00:49:49.2250851Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-10-10T00:49:49.2251157Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-10-10T00:49:49.2251492Z + [[ dynamic_cpu_inductor_huggingface == *operator_microbenchmark* ]] 2025-10-10T00:49:49.2251861Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-10-10T00:49:49.2252162Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-10-10T00:49:49.2252806Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-10-10T00:49:49.2253255Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-10-10T00:49:49.2253561Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-10-10T00:49:49.2253801Z + install_torchvision 2025-10-10T00:49:49.2253978Z + local orig_preload 2025-10-10T00:49:49.2254158Z + local commit 2025-10-10T00:49:49.2254336Z ++ get_pinned_commit vision 2025-10-10T00:49:49.2254546Z ++ cat .github/ci_commit_pins/vision.txt 2025-10-10T00:49:49.2258633Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-10-10T00:49:49.2259006Z + orig_preload= 2025-10-10T00:49:49.2259305Z + '[' -n '' ']' 2025-10-10T00:49:49.2259542Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-10-10T00:49:49.2260153Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision 2025-10-10T00:49:49.2260873Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-10-10T00:49:49.2261280Z + local wheel_dir=dist/vision 2025-10-10T00:49:49.2261499Z + local found_whl=0 2025-10-10T00:49:49.2261701Z + for file in "${wheel_dir}"/*.whl 2025-10-10T00:49:49.2262040Z + [[ -f dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl ]] 2025-10-10T00:49:49.2262350Z + found_whl=1 2025-10-10T00:49:49.2262521Z + break 2025-10-10T00:49:49.2262682Z + '[' 1 == 0 ']' 2025-10-10T00:49:49.2262869Z + for file in "${wheel_dir}"/*.whl 2025-10-10T00:49:49.2263209Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:49.2263670Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl') 2025-10-10T00:49:49.2263991Z + local args 2025-10-10T00:49:49.2264280Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-10-10T00:49:49.2264618Z + for path in "${args[@]}" 2025-10-10T00:49:49.2264951Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl' 2025-10-10T00:49:49.2265413Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:49.2265925Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:49.5345742Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl 2025-10-10T00:49:49.5424934Z Installing collected packages: torchvision 2025-10-10T00:49:50.0012901Z Successfully installed torchvision-0.22.0a0+966da7e 2025-10-10T00:49:50.0462312Z + '[' -n '' ']' 2025-10-10T00:49:50.0463101Z + id=0 2025-10-10T00:49:50.0463378Z + test_dynamo_benchmark huggingface 0 2025-10-10T00:49:50.0463628Z ++ pwd 2025-10-10T00:49:50.0463871Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-10-10T00:49:50.0464218Z + local suite=huggingface 2025-10-10T00:49:50.0464452Z + shift 2025-10-10T00:49:50.0464652Z + local shard_id=0 2025-10-10T00:49:50.0464824Z + shift 2025-10-10T00:49:50.0465080Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-10-10T00:49:50.0465420Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-10-10T00:49:50.0465702Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-10-10T00:49:50.0466012Z + local dt=float32 2025-10-10T00:49:50.0466240Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-10-10T00:49:50.0466522Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-10-10T00:49:50.0466916Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-10-10T00:49:50.0467269Z ++ pwd 2025-10-10T00:49:50.0471751Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-10-10T00:49:50.0472158Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-10-10T00:49:50.0492043Z + local name=inference 2025-10-10T00:49:50.0496101Z + shift 2025-10-10T00:49:50.0500416Z + local suite=huggingface 2025-10-10T00:49:50.0500693Z + shift 2025-10-10T00:49:50.0501040Z + local shard_id=0 2025-10-10T00:49:50.0501231Z + shift 2025-10-10T00:49:50.0501681Z + partition_flags=() 2025-10-10T00:49:50.0501986Z + local partition_flags 2025-10-10T00:49:50.0502173Z + [[ -n 1 ]] 2025-10-10T00:49:50.0502332Z + [[ -n 0 ]] 2025-10-10T00:49:50.0502611Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-10-10T00:49:50.0502980Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-10-10T00:49:50.0503264Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-10-10T00:49:50.0503521Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-10-10T00:49:50.0503781Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-10-10T00:49:50.0504764Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-10-10T00:49:50.8497097Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:49:50.8498046Z import pynvml # type: ignore[import] 2025-10-10T00:49:54.3635687Z 2025-10-10T00:49:54.3636617Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-10-10T00:51:29.1994604Z 2025-10-10T00:51:29.1994721Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.1995291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.1995837Z layer_outputs = layer_module( 2025-10-10T00:51:29.1996224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.1996627Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.1997085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.1997551Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.1998014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.1998471Z self_outputs = self.self( 2025-10-10T00:51:29.1998906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.1999402Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.1999963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2000616Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2000876Z 2025-10-10T00:51:29.2000997Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2001543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2002065Z layer_outputs = layer_module( 2025-10-10T00:51:29.2002447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2002843Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2003294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2003750Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2004203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2004647Z self_outputs = self.self( 2025-10-10T00:51:29.2005079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2005560Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2006088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2006713Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2006982Z 2025-10-10T00:51:29.2007098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2007798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2008339Z layer_outputs = layer_module( 2025-10-10T00:51:29.2008709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2009109Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2009558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2010018Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2010473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2010916Z self_outputs = self.self( 2025-10-10T00:51:29.2011352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2011833Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2012365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2012982Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2013242Z 2025-10-10T00:51:29.2013334Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2013571Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2013799Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2014022Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2014267Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2014827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2015346Z layer_outputs = layer_module( 2025-10-10T00:51:29.2015726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2016120Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2016560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2017018Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2017468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2017930Z self_outputs = self.self( 2025-10-10T00:51:29.2018365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2018849Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2019422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2020010Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2020591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2021122Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2021292Z 2025-10-10T00:51:29.2021381Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2021645Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2022215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2022747Z layer_outputs = layer_module( 2025-10-10T00:51:29.2023223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2023627Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2024093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2024558Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2025021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2025472Z self_outputs = self.self( 2025-10-10T00:51:29.2025917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2026489Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2026633Z 2025-10-10T00:51:29.2026767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2027355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2027900Z layer_outputs = layer_module( 2025-10-10T00:51:29.2028304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2028722Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2029196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2029669Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2030131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2030607Z self_outputs = self.self( 2025-10-10T00:51:29.2031064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2031694Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2031861Z 2025-10-10T00:51:29.2031986Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2032539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2033063Z layer_outputs = layer_module( 2025-10-10T00:51:29.2033441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2033837Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2034286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2034748Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2035206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2035646Z self_outputs = self.self( 2025-10-10T00:51:29.2036077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2036578Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2037151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2037780Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2038242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2038628Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2038793Z 2025-10-10T00:51:29.2039060Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2039616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2040142Z layer_outputs = layer_module( 2025-10-10T00:51:29.2040523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2040919Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2041368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2041821Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2042269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2042715Z self_outputs = self.self( 2025-10-10T00:51:29.2043153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2043641Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2044211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2044802Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2045359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2045866Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2046234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2046622Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2046796Z 2025-10-10T00:51:29.2046916Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2047473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2047995Z layer_outputs = layer_module( 2025-10-10T00:51:29.2048370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2048767Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2049233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2049700Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2050169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2050616Z self_outputs = self.self( 2025-10-10T00:51:29.2051047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2051552Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2052118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2052723Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2052948Z 2025-10-10T00:51:29.2053063Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2053618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2054148Z layer_outputs = layer_module( 2025-10-10T00:51:29.2054581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2055911Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2056364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2056823Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2057275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2057728Z self_outputs = self.self( 2025-10-10T00:51:29.2058186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2058675Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2059250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2059858Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2060080Z 2025-10-10T00:51:29.2060201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2060762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2061276Z layer_outputs = layer_module( 2025-10-10T00:51:29.2061650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2062044Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2062497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2062948Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2063401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2063850Z self_outputs = self.self( 2025-10-10T00:51:29.2064297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2064892Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2065161Z 2025-10-10T00:51:29.2065286Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2065855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2066506Z layer_outputs = layer_module( 2025-10-10T00:51:29.2066901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2067316Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2067780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2068240Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2068697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2069200Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2069704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2070178Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2070340Z 2025-10-10T00:51:29.2070454Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2071086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2071640Z layer_outputs = layer_module( 2025-10-10T00:51:29.2072018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2072404Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2072854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2073316Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2073760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2074201Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2074659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2075162Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2075646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2076114Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2076263Z 2025-10-10T00:51:29.2076384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2076928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2077464Z layer_outputs = layer_module( 2025-10-10T00:51:29.2077852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2078257Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2078733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2079197Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2079641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2080096Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2080539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2081023Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2081513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2082015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2082442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2082834Z return self.act(input) 2025-10-10T00:51:29.2082959Z 2025-10-10T00:51:29.2083074Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2083640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2084176Z layer_outputs = layer_module( 2025-10-10T00:51:29.2084565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2084974Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2085442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2085924Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2086424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2086908Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2087367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2087881Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2088398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2088881Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2089035Z 2025-10-10T00:51:29.2089160Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2089728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2090267Z layer_outputs = layer_module( 2025-10-10T00:51:29.2090666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2091070Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2091535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2092010Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2092499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2092954Z self_outputs = self.self( 2025-10-10T00:51:29.2093399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2093870Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2094022Z 2025-10-10T00:51:29.2094137Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2094708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2095241Z layer_outputs = layer_module( 2025-10-10T00:51:29.2095629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2096035Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2096495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2096973Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2097441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2097909Z self_outputs = self.self( 2025-10-10T00:51:29.2098354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2098845Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2099402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2100050Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2100322Z 2025-10-10T00:51:29.2100448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2101016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2101545Z layer_outputs = layer_module( 2025-10-10T00:51:29.2101930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2102424Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2102880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2103331Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2103785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2104243Z self_outputs = self.self( 2025-10-10T00:51:29.2104688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2105156Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2105306Z 2025-10-10T00:51:29.2105430Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2106086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2106639Z layer_outputs = layer_module( 2025-10-10T00:51:29.2107029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2107435Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2107905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2108379Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2108836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2109287Z self_outputs = self.self( 2025-10-10T00:51:29.2109726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2110207Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2110771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2111411Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2111686Z 2025-10-10T00:51:29.2111801Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2112360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2112896Z layer_outputs = layer_module( 2025-10-10T00:51:29.2113271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2113673Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2114131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2114591Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2115049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2115508Z self_outputs = self.self( 2025-10-10T00:51:29.2115948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2116440Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2116988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2117619Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2117888Z 2025-10-10T00:51:29.2118003Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2118661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2119182Z layer_outputs = layer_module( 2025-10-10T00:51:29.2119558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2119944Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2120399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2120853Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2121305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2121755Z self_outputs = self.self( 2025-10-10T00:51:29.2122178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2122660Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2123192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2123815Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2124070Z 2025-10-10T00:51:29.2124170Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2124399Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2124630Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2124853Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2125105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2125657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2126195Z layer_outputs = layer_module( 2025-10-10T00:51:29.2126583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2126990Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2127459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2127918Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2128380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2128837Z self_outputs = self.self( 2025-10-10T00:51:29.2129280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2129782Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2130333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2130935Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2131671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2132227Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2132397Z 2025-10-10T00:51:29.2132495Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2132756Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2133334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2134021Z layer_outputs = layer_module( 2025-10-10T00:51:29.2134423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2134838Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2135313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2135806Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2136291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2136771Z self_outputs = self.self( 2025-10-10T00:51:29.2137212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2137687Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2137836Z 2025-10-10T00:51:29.2137956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2138534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2139075Z layer_outputs = layer_module( 2025-10-10T00:51:29.2139462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2139880Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2140350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2140835Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2141313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2141786Z self_outputs = self.self( 2025-10-10T00:51:29.2142236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2142712Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2142871Z 2025-10-10T00:51:29.2142993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2143552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2144075Z layer_outputs = layer_module( 2025-10-10T00:51:29.2144459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2144859Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2145317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2145774Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2146298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2146756Z self_outputs = self.self( 2025-10-10T00:51:29.2147205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2147657Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2147808Z 2025-10-10T00:51:29.2147931Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2148472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2148993Z layer_outputs = layer_module( 2025-10-10T00:51:29.2149378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2149861Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2150310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2150767Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2151224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2151675Z self_outputs = self.self( 2025-10-10T00:51:29.2152114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2152613Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2153192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2153842Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2154316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2154712Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2154877Z 2025-10-10T00:51:29.2154991Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2155555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2156084Z layer_outputs = layer_module( 2025-10-10T00:51:29.2156468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2156868Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2157317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2157783Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2158236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2158664Z self_outputs = self.self( 2025-10-10T00:51:29.2159075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2159546Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2160088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2160650Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2161185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2161682Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2162050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2162436Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2162608Z 2025-10-10T00:51:29.2162723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2163287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2163809Z layer_outputs = layer_module( 2025-10-10T00:51:29.2164165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2164542Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2165030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2165512Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2165965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2166397Z self_outputs = self.self( 2025-10-10T00:51:29.2166810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2167309Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2167874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2168478Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2168701Z 2025-10-10T00:51:29.2168813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2169378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2169899Z layer_outputs = layer_module( 2025-10-10T00:51:29.2170277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2170675Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2171124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2171578Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2172033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2172490Z self_outputs = self.self( 2025-10-10T00:51:29.2172932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2173484Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2174052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2174656Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2174875Z 2025-10-10T00:51:29.2174995Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2175545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2176070Z layer_outputs = layer_module( 2025-10-10T00:51:29.2176448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2176849Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2177330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2177806Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2178255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2178697Z self_outputs = self.self( 2025-10-10T00:51:29.2179130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2179705Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2179961Z 2025-10-10T00:51:29.2180073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2180683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2181260Z layer_outputs = layer_module( 2025-10-10T00:51:29.2181639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2182035Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2182494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2182965Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2183433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2183944Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2184444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2184908Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2185073Z 2025-10-10T00:51:29.2185198Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2185744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2186338Z layer_outputs = layer_module( 2025-10-10T00:51:29.2186716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2187102Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2187554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2188017Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2188462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2188903Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2189347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2189844Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2190331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2190787Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2190937Z 2025-10-10T00:51:29.2191058Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2191603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2192125Z layer_outputs = layer_module( 2025-10-10T00:51:29.2192514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2192910Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2193354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2193814Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2194253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2194686Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2195140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2195624Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2196146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2196662Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2197079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2197464Z return self.act(input) 2025-10-10T00:51:29.2197590Z 2025-10-10T00:51:29.2197716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2198268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2198787Z layer_outputs = layer_module( 2025-10-10T00:51:29.2199162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2199553Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2200003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2200461Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2200898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2201336Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2201777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2202270Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2202765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2203231Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2203379Z 2025-10-10T00:51:29.2203501Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2204049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2204562Z layer_outputs = layer_module( 2025-10-10T00:51:29.2204936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2205326Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2205774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2206228Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2206691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2207145Z self_outputs = self.self( 2025-10-10T00:51:29.2207590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2208077Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2208224Z 2025-10-10T00:51:29.2208337Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2208881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2209407Z layer_outputs = layer_module( 2025-10-10T00:51:29.2209799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2210209Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2210661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2211135Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2211711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2212187Z self_outputs = self.self( 2025-10-10T00:51:29.2212644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2213144Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2213749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2214395Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2214667Z 2025-10-10T00:51:29.2214794Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2215373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2215918Z layer_outputs = layer_module( 2025-10-10T00:51:29.2216306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2216717Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2217186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2217648Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2218103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2218571Z self_outputs = self.self( 2025-10-10T00:51:29.2219019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2219492Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2219643Z 2025-10-10T00:51:29.2219766Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2220329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2220865Z layer_outputs = layer_module( 2025-10-10T00:51:29.2221249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2221652Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2222109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2222575Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2223042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2223508Z self_outputs = self.self( 2025-10-10T00:51:29.2223950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2224445Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2225003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2225648Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2225916Z 2025-10-10T00:51:29.2226115Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2226687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2227213Z layer_outputs = layer_module( 2025-10-10T00:51:29.2227684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2228099Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2228569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2229034Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2229488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2229948Z self_outputs = self.self( 2025-10-10T00:51:29.2230396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2230889Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2231448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2232183Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2232451Z 2025-10-10T00:51:29.2232568Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2233122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2233644Z layer_outputs = layer_module( 2025-10-10T00:51:29.2234025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2234416Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2234872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2235336Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2235792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2236237Z self_outputs = self.self( 2025-10-10T00:51:29.2236664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2237158Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2237711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2238333Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2238590Z 2025-10-10T00:51:29.2238688Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2238916Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2239153Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2239390Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2239632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2240151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2240642Z layer_outputs = layer_module( 2025-10-10T00:51:29.2241000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2241393Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2241844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2242263Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2242786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2243260Z self_outputs = self.self( 2025-10-10T00:51:29.2243666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2244121Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2244627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2245180Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2245737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2246251Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2246417Z 2025-10-10T00:51:29.2246514Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2246772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2247324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2247844Z layer_outputs = layer_module( 2025-10-10T00:51:29.2248228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2248628Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2249091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2249551Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2250005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2250459Z self_outputs = self.self( 2025-10-10T00:51:29.2250891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2251345Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2251485Z 2025-10-10T00:51:29.2251596Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2252142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2252657Z layer_outputs = layer_module( 2025-10-10T00:51:29.2253025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2253419Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2253880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2254430Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2254889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2255320Z self_outputs = self.self( 2025-10-10T00:51:29.2255750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2256201Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2256345Z 2025-10-10T00:51:29.2256465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2257015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2257527Z layer_outputs = layer_module( 2025-10-10T00:51:29.2257903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2258379Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2258835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2259291Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2259743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2260196Z self_outputs = self.self( 2025-10-10T00:51:29.2260645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2261117Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2261273Z 2025-10-10T00:51:29.2261389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2261965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2262509Z layer_outputs = layer_module( 2025-10-10T00:51:29.2262888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2263285Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2263731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2264248Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2264721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2265191Z self_outputs = self.self( 2025-10-10T00:51:29.2265635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2266213Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2266809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2267453Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2267928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2268320Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2268493Z 2025-10-10T00:51:29.2268611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2269181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2269719Z layer_outputs = layer_module( 2025-10-10T00:51:29.2270116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2270520Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2270975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2271437Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2271896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2272350Z self_outputs = self.self( 2025-10-10T00:51:29.2272785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2273283Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2273919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2274561Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2275129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2275648Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2276022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2276415Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2276592Z 2025-10-10T00:51:29.2276711Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2277280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2277818Z layer_outputs = layer_module( 2025-10-10T00:51:29.2278209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2278618Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2279089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2279538Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2279986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2280431Z self_outputs = self.self( 2025-10-10T00:51:29.2280857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2281344Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2281908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2282502Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2282729Z 2025-10-10T00:51:29.2282843Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2283397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2283919Z layer_outputs = layer_module( 2025-10-10T00:51:29.2284300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2284686Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2285136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2285594Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2286054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2286511Z self_outputs = self.self( 2025-10-10T00:51:29.2286946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2287452Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2288037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2288639Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2288858Z 2025-10-10T00:51:29.2288980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2289584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2290158Z layer_outputs = layer_module( 2025-10-10T00:51:29.2290548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2290952Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2291415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2291871Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2292334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2292785Z self_outputs = self.self( 2025-10-10T00:51:29.2293230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2293819Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2294083Z 2025-10-10T00:51:29.2294201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2294773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2295305Z layer_outputs = layer_module( 2025-10-10T00:51:29.2295694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2296101Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2296558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2297024Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2297490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2297993Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2298496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2298960Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2299123Z 2025-10-10T00:51:29.2299240Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2299809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2300348Z layer_outputs = layer_module( 2025-10-10T00:51:29.2300735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2301134Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2301602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2302074Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2302526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2302963Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2303429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2303936Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2304433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2304903Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2305113Z 2025-10-10T00:51:29.2305262Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2305833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2306446Z layer_outputs = layer_module( 2025-10-10T00:51:29.2306839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2307247Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2307703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2308174Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2308627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2309072Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2309544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2310044Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2310538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2311041Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2311467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2311855Z return self.act(input) 2025-10-10T00:51:29.2311991Z 2025-10-10T00:51:29.2312105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2312658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2313209Z layer_outputs = layer_module( 2025-10-10T00:51:29.2313591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2314004Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2314443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2314894Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2315336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2315772Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2316222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2316739Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2317256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2317727Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2317876Z 2025-10-10T00:51:29.2317996Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2318548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2319082Z layer_outputs = layer_module( 2025-10-10T00:51:29.2319475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2319886Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2320370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2320894Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2321347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2321796Z self_outputs = self.self( 2025-10-10T00:51:29.2322234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2322686Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2322834Z 2025-10-10T00:51:29.2322947Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2323499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2324018Z layer_outputs = layer_module( 2025-10-10T00:51:29.2324395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2324792Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2325236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2325684Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2326130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2326577Z self_outputs = self.self( 2025-10-10T00:51:29.2327020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2327520Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2328073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2328730Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2329004Z 2025-10-10T00:51:29.2329129Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2329696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2330225Z layer_outputs = layer_module( 2025-10-10T00:51:29.2330609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2331013Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2331660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2332133Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2332609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2333082Z self_outputs = self.self( 2025-10-10T00:51:29.2333539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2334018Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2334171Z 2025-10-10T00:51:29.2334289Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2334877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2335430Z layer_outputs = layer_module( 2025-10-10T00:51:29.2335826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2336242Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2336789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2337320Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2337782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2338245Z self_outputs = self.self( 2025-10-10T00:51:29.2338688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2339175Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2339735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2340383Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2340657Z 2025-10-10T00:51:29.2341260Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2341841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2342364Z layer_outputs = layer_module( 2025-10-10T00:51:29.2342739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2343133Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2343585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2344045Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2344494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2344957Z self_outputs = self.self( 2025-10-10T00:51:29.2345394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2345882Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2346512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2347151Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2347423Z 2025-10-10T00:51:29.2347540Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2348109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2348637Z layer_outputs = layer_module( 2025-10-10T00:51:29.2349017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2349408Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2349858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2350310Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2350761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2351210Z self_outputs = self.self( 2025-10-10T00:51:29.2351632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2352104Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2352634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2353358Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2353615Z 2025-10-10T00:51:29.2353715Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2353943Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2354171Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2354397Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2354649Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2355198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2355864Z layer_outputs = layer_module( 2025-10-10T00:51:29.2356301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2357011Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2357509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2358029Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2358612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2359142Z self_outputs = self.self( 2025-10-10T00:51:29.2359626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2360147Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2360761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2361430Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2362093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2362682Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2362858Z 2025-10-10T00:51:29.2362997Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2363339Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2363974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2364587Z layer_outputs = layer_module( 2025-10-10T00:51:29.2365024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2365450Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2365963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2366458Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2366791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2366916Z self_outputs = self.self( 2025-10-10T00:51:29.2367268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2367418Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2367423Z 2025-10-10T00:51:29.2367560Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2368009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2368114Z layer_outputs = layer_module( 2025-10-10T00:51:29.2368410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2368623Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2368927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2369065Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2369370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2369484Z self_outputs = self.self( 2025-10-10T00:51:29.2369807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2369945Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2369950Z 2025-10-10T00:51:29.2370121Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2370522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2370660Z layer_outputs = layer_module( 2025-10-10T00:51:29.2370908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2371111Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2371448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2371552Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2371914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2372011Z self_outputs = self.self( 2025-10-10T00:51:29.2391590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2391864Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2391871Z 2025-10-10T00:51:29.2392020Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2392464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2392563Z layer_outputs = layer_module( 2025-10-10T00:51:29.2392832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2392934Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2393254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2393356Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2393671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2393759Z self_outputs = self.self( 2025-10-10T00:51:29.2394081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2394219Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2394614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2394809Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2395026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2395150Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2395155Z 2025-10-10T00:51:29.2395275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2395877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2395963Z layer_outputs = layer_module( 2025-10-10T00:51:29.2396219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2396313Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2396624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2396724Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2397039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2397134Z self_outputs = self.self( 2025-10-10T00:51:29.2397451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2397597Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2397992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2398148Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2398516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2398624Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2398850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2398959Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2398963Z 2025-10-10T00:51:29.2399090Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2399494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2399580Z layer_outputs = layer_module( 2025-10-10T00:51:29.2399840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2399931Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2400246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2400335Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2400645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2400733Z self_outputs = self.self( 2025-10-10T00:51:29.2401049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2401186Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2401574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2401757Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2401761Z 2025-10-10T00:51:29.2401883Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2402274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2402364Z layer_outputs = layer_module( 2025-10-10T00:51:29.2402646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2402780Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2403094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2403183Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2403501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2403579Z self_outputs = self.self( 2025-10-10T00:51:29.2403897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2404033Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2404430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2404604Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2404609Z 2025-10-10T00:51:29.2404726Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2405125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2405207Z layer_outputs = layer_module( 2025-10-10T00:51:29.2405461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2405551Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2405871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2405957Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2406272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2406366Z self_outputs = self.self( 2025-10-10T00:51:29.2406678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2406898Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2406903Z 2025-10-10T00:51:29.2407018Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2407418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2407498Z layer_outputs = layer_module( 2025-10-10T00:51:29.2407743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2407849Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2408160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2408253Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2408560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2408689Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2409010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2409107Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2409111Z 2025-10-10T00:51:29.2409236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2409656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2409780Z layer_outputs = layer_module( 2025-10-10T00:51:29.2410026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2410114Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2410438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2410535Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2410839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2410928Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2411247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2411387Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2411702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2411804Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2411807Z 2025-10-10T00:51:29.2411921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2412312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2412391Z layer_outputs = layer_module( 2025-10-10T00:51:29.2412630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2412723Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2413033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2413137Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2413426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2413522Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2413828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2413948Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2414263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2414388Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2414634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2414717Z return self.act(input) 2025-10-10T00:51:29.2414724Z 2025-10-10T00:51:29.2414838Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2415226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2415308Z layer_outputs = layer_module( 2025-10-10T00:51:29.2415564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2415653Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2415980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2416072Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2416354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2416528Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2416846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2416994Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2417309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2417402Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2417415Z 2025-10-10T00:51:29.2417533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2417925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2418019Z layer_outputs = layer_module( 2025-10-10T00:51:29.2418269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2418371Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2418682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2418770Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2419092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2419172Z self_outputs = self.self( 2025-10-10T00:51:29.2419489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2419583Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2419587Z 2025-10-10T00:51:29.2419710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2420103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2420188Z layer_outputs = layer_module( 2025-10-10T00:51:29.2420443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2420531Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2420846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2420933Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2421246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2421332Z self_outputs = self.self( 2025-10-10T00:51:29.2421644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2421773Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2422156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2422375Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2422379Z 2025-10-10T00:51:29.2422496Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2422898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2422986Z layer_outputs = layer_module( 2025-10-10T00:51:29.2423240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2423335Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2423716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2423810Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2424125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2424204Z self_outputs = self.self( 2025-10-10T00:51:29.2424527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2424618Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2424622Z 2025-10-10T00:51:29.2424746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2425132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2425220Z layer_outputs = layer_module( 2025-10-10T00:51:29.2425476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2425566Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2425882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2426232Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2426561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2426642Z self_outputs = self.self( 2025-10-10T00:51:29.2426951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2427076Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2427476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2427688Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2427692Z 2025-10-10T00:51:29.2427805Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2428191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2428273Z layer_outputs = layer_module( 2025-10-10T00:51:29.2428518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2428617Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2428943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2429038Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2429341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2429419Z self_outputs = self.self( 2025-10-10T00:51:29.2429728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2429841Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2430217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2430417Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2430421Z 2025-10-10T00:51:29.2430543Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2431020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2431101Z layer_outputs = layer_module( 2025-10-10T00:51:29.2431345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2431431Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2432060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2432148Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2432457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2432533Z self_outputs = self.self( 2025-10-10T00:51:29.2432836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2432959Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2433325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2433528Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2433532Z 2025-10-10T00:51:29.2433626Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2433718Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2433815Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2433900Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2434023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2434405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2434498Z layer_outputs = layer_module( 2025-10-10T00:51:29.2434740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2434828Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2435137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2435221Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2435533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2435614Z self_outputs = self.self( 2025-10-10T00:51:29.2435915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2436047Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2436416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2436581Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2436933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2437051Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2437055Z 2025-10-10T00:51:29.2437145Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2437262Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2437659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2437741Z layer_outputs = layer_module( 2025-10-10T00:51:29.2438161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2438252Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2438552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2438644Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2438943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2439030Z self_outputs = self.self( 2025-10-10T00:51:29.2439331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2439423Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2439427Z 2025-10-10T00:51:29.2439540Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2439924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2440014Z layer_outputs = layer_module( 2025-10-10T00:51:29.2440252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2440345Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2440648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2440735Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2441035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2441112Z self_outputs = self.self( 2025-10-10T00:51:29.2441426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2441517Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2441521Z 2025-10-10T00:51:29.2441642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2442012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2442091Z layer_outputs = layer_module( 2025-10-10T00:51:29.2442338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2442424Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2442744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2442830Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2443156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2443246Z self_outputs = self.self( 2025-10-10T00:51:29.2443545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2443645Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2443649Z 2025-10-10T00:51:29.2443764Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2444152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2444232Z layer_outputs = layer_module( 2025-10-10T00:51:29.2444472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2444566Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2444962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2445056Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2445358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2445445Z self_outputs = self.self( 2025-10-10T00:51:29.2445747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2445877Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2446265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2446466Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2446703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2446820Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2446823Z 2025-10-10T00:51:29.2446949Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2447344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2447424Z layer_outputs = layer_module( 2025-10-10T00:51:29.2447678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2447766Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2448087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2448180Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2448497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2448585Z self_outputs = self.self( 2025-10-10T00:51:29.2448890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2449028Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2449411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2449570Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2449917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2450027Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2450251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2450363Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2450367Z 2025-10-10T00:51:29.2450489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2450882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2450971Z layer_outputs = layer_module( 2025-10-10T00:51:29.2451211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2451299Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2451614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2451764Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2452073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2452151Z self_outputs = self.self( 2025-10-10T00:51:29.2452449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2452584Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2452958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2453131Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2453136Z 2025-10-10T00:51:29.2453250Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2453641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2453720Z layer_outputs = layer_module( 2025-10-10T00:51:29.2453959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2454054Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2454357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2454449Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2454759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2454844Z self_outputs = self.self( 2025-10-10T00:51:29.2455144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2455269Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2455651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2455815Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2455820Z 2025-10-10T00:51:29.2455940Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2456313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2456402Z layer_outputs = layer_module( 2025-10-10T00:51:29.2456646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2456740Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2457069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2457151Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2457466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2457547Z self_outputs = self.self( 2025-10-10T00:51:29.2457842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2458051Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2458055Z 2025-10-10T00:51:29.2458165Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2458580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2458690Z layer_outputs = layer_module( 2025-10-10T00:51:29.2458938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2459025Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2459344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2459435Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2459746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2459878Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2460201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2460304Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2460316Z 2025-10-10T00:51:29.2460431Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2460829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2460919Z layer_outputs = layer_module( 2025-10-10T00:51:29.2461169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2461264Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2461583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2461678Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2461995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2462087Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2462420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2462547Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2462874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2462969Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2462973Z 2025-10-10T00:51:29.2463088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2463494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2463575Z layer_outputs = layer_module( 2025-10-10T00:51:29.2463840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2463928Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2464253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2464354Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2464662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2464749Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2465070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2465198Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2465548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2465716Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2466043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2466140Z return self.act(input) 2025-10-10T00:51:29.2466145Z 2025-10-10T00:51:29.2466270Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2466657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2466749Z layer_outputs = layer_module( 2025-10-10T00:51:29.2466991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2467088Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2467401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2467498Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2467793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2467878Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2468201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2468339Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2468664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2468756Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2468760Z 2025-10-10T00:51:29.2468872Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2469266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2469344Z layer_outputs = layer_module( 2025-10-10T00:51:29.2469589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2469677Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2469984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2470081Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2470390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2470479Z self_outputs = self.self( 2025-10-10T00:51:29.2470794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2470896Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2470900Z 2025-10-10T00:51:29.2471014Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2471397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2471485Z layer_outputs = layer_module( 2025-10-10T00:51:29.2471728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2471822Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2472129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2472222Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2472617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2472699Z self_outputs = self.self( 2025-10-10T00:51:29.2473012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2473126Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2473512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2473716Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2473720Z 2025-10-10T00:51:29.2473835Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2474230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2474314Z layer_outputs = layer_module( 2025-10-10T00:51:29.2474566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2474652Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2474967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2475050Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2475368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2475455Z self_outputs = self.self( 2025-10-10T00:51:29.2475760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2475860Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2475869Z 2025-10-10T00:51:29.2475984Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2476379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2476458Z layer_outputs = layer_module( 2025-10-10T00:51:29.2476702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2476795Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2477106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2477197Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2477507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2477590Z self_outputs = self.self( 2025-10-10T00:51:29.2477904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2478016Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2478409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2478606Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2478609Z 2025-10-10T00:51:29.2478728Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2479104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2479181Z layer_outputs = layer_module( 2025-10-10T00:51:29.2479492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2479580Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2479888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2479972Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2480277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2480354Z self_outputs = self.self( 2025-10-10T00:51:29.2480653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2480771Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2481146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2481350Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2481354Z 2025-10-10T00:51:29.2481465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2481848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2481926Z layer_outputs = layer_module( 2025-10-10T00:51:29.2482163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2482255Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2482555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2482644Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2482950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2483026Z self_outputs = self.self( 2025-10-10T00:51:29.2483332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2483439Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2483810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2484003Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2484006Z 2025-10-10T00:51:29.2484103Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2484189Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2484271Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2484368Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2484478Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2484862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2484938Z layer_outputs = layer_module( 2025-10-10T00:51:29.2485176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2485268Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2485573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2485661Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2486000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2486113Z self_outputs = self.self( 2025-10-10T00:51:29.2486435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2486559Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2486955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2487116Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2487489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2487597Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2487601Z 2025-10-10T00:51:29.2487689Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2487817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2488217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2488307Z layer_outputs = layer_module( 2025-10-10T00:51:29.2488567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2488662Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2488972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2489054Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2489371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2489448Z self_outputs = self.self( 2025-10-10T00:51:29.2489770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2489850Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2489854Z 2025-10-10T00:51:29.2489966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2490362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2490441Z layer_outputs = layer_module( 2025-10-10T00:51:29.2490690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2490774Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2491090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2491173Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2491491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2491575Z self_outputs = self.self( 2025-10-10T00:51:29.2491882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2491977Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2491981Z 2025-10-10T00:51:29.2492092Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2492488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2492566Z layer_outputs = layer_module( 2025-10-10T00:51:29.2492809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2492987Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2493288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2493377Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2493677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2493753Z self_outputs = self.self( 2025-10-10T00:51:29.2494060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2494153Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2494157Z 2025-10-10T00:51:29.2494277Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2494655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2494742Z layer_outputs = layer_module( 2025-10-10T00:51:29.2494979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2495066Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2495376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2495457Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2495768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2495845Z self_outputs = self.self( 2025-10-10T00:51:29.2496144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2496285Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2496681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2496881Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2497099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2497220Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2497224Z 2025-10-10T00:51:29.2497338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2497727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2497816Z layer_outputs = layer_module( 2025-10-10T00:51:29.2498064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2498173Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2498472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2498563Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2498865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2498941Z self_outputs = self.self( 2025-10-10T00:51:29.2499258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2499387Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2499820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2500006Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2500355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2500466Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2500678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2500793Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2500797Z 2025-10-10T00:51:29.2500912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2501304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2501383Z layer_outputs = layer_module( 2025-10-10T00:51:29.2501633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2501730Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2502039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2502130Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2502439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2502517Z self_outputs = self.self( 2025-10-10T00:51:29.2502831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2502960Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2503357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2503530Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2503535Z 2025-10-10T00:51:29.2503658Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2504046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2504125Z layer_outputs = layer_module( 2025-10-10T00:51:29.2504376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2504465Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2504778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2504863Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2505188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2505266Z self_outputs = self.self( 2025-10-10T00:51:29.2505571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2505707Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2506190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2506369Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2506374Z 2025-10-10T00:51:29.2506487Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2506930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2507051Z layer_outputs = layer_module( 2025-10-10T00:51:29.2507293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2507391Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2507701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2507794Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2508104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2508182Z self_outputs = self.self( 2025-10-10T00:51:29.2508498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2508712Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2508716Z 2025-10-10T00:51:29.2508840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2509235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2509324Z layer_outputs = layer_module( 2025-10-10T00:51:29.2509567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2509654Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2509969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2510052Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2510367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2510498Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2510815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2510908Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2510912Z 2025-10-10T00:51:29.2511026Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2511432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2511511Z layer_outputs = layer_module( 2025-10-10T00:51:29.2511760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2511846Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2512161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2512263Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2512554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2512647Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2512960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2513091Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2513399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2513492Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2513495Z 2025-10-10T00:51:29.2513687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2514078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2514166Z layer_outputs = layer_module( 2025-10-10T00:51:29.2514412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2514506Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2514816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2514909Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2515210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2515298Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2515629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2515750Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2516062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2516198Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2516439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2516530Z return self.act(input) 2025-10-10T00:51:29.2516534Z 2025-10-10T00:51:29.2516656Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2517044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2517141Z layer_outputs = layer_module( 2025-10-10T00:51:29.2517385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2517474Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2517794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2517888Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2518183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2518291Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2518608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2518754Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2519071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2519170Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2519174Z 2025-10-10T00:51:29.2519289Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2519684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2519772Z layer_outputs = layer_module( 2025-10-10T00:51:29.2520019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2520115Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2520426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2520520Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2521453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2521546Z self_outputs = self.self( 2025-10-10T00:51:29.2521867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2521960Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2521964Z 2025-10-10T00:51:29.2522087Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2522479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2522560Z layer_outputs = layer_module( 2025-10-10T00:51:29.2522813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2522908Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2523222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2523306Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2523621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2523700Z self_outputs = self.self( 2025-10-10T00:51:29.2524009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2524132Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2524505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2524721Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2524728Z 2025-10-10T00:51:29.2524844Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2525239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2525319Z layer_outputs = layer_module( 2025-10-10T00:51:29.2525563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2525657Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2525966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2526058Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2526366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2526446Z self_outputs = self.self( 2025-10-10T00:51:29.2526759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2526848Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2526852Z 2025-10-10T00:51:29.2526971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2527356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2527442Z layer_outputs = layer_module( 2025-10-10T00:51:29.2527685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2527771Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2528129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2528250Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2528564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2528640Z self_outputs = self.self( 2025-10-10T00:51:29.2528946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2529065Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2529440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2529648Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2529652Z 2025-10-10T00:51:29.2529769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2530165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2530245Z layer_outputs = layer_module( 2025-10-10T00:51:29.2530487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2530582Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2530888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2530980Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2531285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2531370Z self_outputs = self.self( 2025-10-10T00:51:29.2531919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2532042Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2532428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2532629Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2532632Z 2025-10-10T00:51:29.2532756Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2533148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2533234Z layer_outputs = layer_module( 2025-10-10T00:51:29.2533470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2533563Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2533871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2533954Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2534261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2534339Z self_outputs = self.self( 2025-10-10T00:51:29.2534642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2534761Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2535123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2535465Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2535469Z 2025-10-10T00:51:29.2535561Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2535655Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2535739Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2535823Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2535943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2536348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2536434Z layer_outputs = layer_module( 2025-10-10T00:51:29.2536671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2536759Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2537074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2537162Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2537468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2537547Z self_outputs = self.self( 2025-10-10T00:51:29.2537858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2537980Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2538362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2538524Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2538869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2538984Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2538988Z 2025-10-10T00:51:29.2539073Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2539184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2539569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2539647Z layer_outputs = layer_module( 2025-10-10T00:51:29.2539891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2539978Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2540284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2540372Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2540670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2540756Z self_outputs = self.self( 2025-10-10T00:51:29.2541051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2541138Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2541141Z 2025-10-10T00:51:29.2541253Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2541633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2541713Z layer_outputs = layer_module( 2025-10-10T00:51:29.2542001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2542128Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2542429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2542518Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2542819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2542898Z self_outputs = self.self( 2025-10-10T00:51:29.2543209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2543299Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2543303Z 2025-10-10T00:51:29.2543435Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2543810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2543895Z layer_outputs = layer_module( 2025-10-10T00:51:29.2544130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2544214Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2544519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2544600Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2544912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2544988Z self_outputs = self.self( 2025-10-10T00:51:29.2545295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2545403Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2545408Z 2025-10-10T00:51:29.2545519Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2545910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2546055Z layer_outputs = layer_module( 2025-10-10T00:51:29.2546314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2546401Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2546709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2546800Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2547113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2547200Z self_outputs = self.self( 2025-10-10T00:51:29.2547519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2547648Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2548035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2548223Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2548444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2548553Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2548556Z 2025-10-10T00:51:29.2548677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2549160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2549241Z layer_outputs = layer_module( 2025-10-10T00:51:29.2549489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2549574Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2549883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2549964Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2550269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2550346Z self_outputs = self.self( 2025-10-10T00:51:29.2550654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2550793Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2551191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2551341Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2551673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2551776Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2551978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2552080Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2552083Z 2025-10-10T00:51:29.2552201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2552568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2552652Z layer_outputs = layer_module( 2025-10-10T00:51:29.2552880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2552963Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2553261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2553341Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2553631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2553703Z self_outputs = self.self( 2025-10-10T00:51:29.2554003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2554124Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2554502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2554677Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2554681Z 2025-10-10T00:51:29.2554790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2555187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2555266Z layer_outputs = layer_module( 2025-10-10T00:51:29.2555519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2555678Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2555980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2556070Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2556368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2556451Z self_outputs = self.self( 2025-10-10T00:51:29.2556753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2556875Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2557274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2557441Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2557445Z 2025-10-10T00:51:29.2557565Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2557945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2558029Z layer_outputs = layer_module( 2025-10-10T00:51:29.2558272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2558353Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2558643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2558722Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2559018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2559092Z self_outputs = self.self( 2025-10-10T00:51:29.2559429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2559631Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2559635Z 2025-10-10T00:51:29.2559746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2560145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2560222Z layer_outputs = layer_module( 2025-10-10T00:51:29.2560465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2560552Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2560858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2560948Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2561248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2561377Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2561676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2561776Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2561780Z 2025-10-10T00:51:29.2561888Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2562308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2562424Z layer_outputs = layer_module( 2025-10-10T00:51:29.2562658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2562749Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2563046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2563144Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2563423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2563507Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2563816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2563935Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2564244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2564334Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2564338Z 2025-10-10T00:51:29.2564456Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2564814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2564888Z layer_outputs = layer_module( 2025-10-10T00:51:29.2565117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2565196Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2565481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2565571Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2565844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2565934Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2566233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2566356Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2566649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2566771Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2566990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2567064Z return self.act(input) 2025-10-10T00:51:29.2567070Z 2025-10-10T00:51:29.2567187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2567589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2567674Z layer_outputs = layer_module( 2025-10-10T00:51:29.2567897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2567979Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2568265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2568352Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2568621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2568699Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2569051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2569182Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2569462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2569555Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2569558Z 2025-10-10T00:51:29.2569661Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2570023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2570097Z layer_outputs = layer_module( 2025-10-10T00:51:29.2570334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2570430Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2570728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2570820Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2571118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2571201Z self_outputs = self.self( 2025-10-10T00:51:29.2571499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2571589Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2571593Z 2025-10-10T00:51:29.2571711Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2572089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2572177Z layer_outputs = layer_module( 2025-10-10T00:51:29.2572411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2572504Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2572801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2572880Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2573168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2573240Z self_outputs = self.self( 2025-10-10T00:51:29.2573529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2573637Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2573980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2574176Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2574179Z 2025-10-10T00:51:29.2574286Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2574645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2574717Z layer_outputs = layer_module( 2025-10-10T00:51:29.2574947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2575030Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2575390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2575476Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2575759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2575837Z self_outputs = self.self( 2025-10-10T00:51:29.2576121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2576204Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2576215Z 2025-10-10T00:51:29.2576320Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2576706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2576789Z layer_outputs = layer_module( 2025-10-10T00:51:29.2577033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2577127Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2577426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2577508Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2577816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2577890Z self_outputs = self.self( 2025-10-10T00:51:29.2578210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2578312Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2578672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2578862Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2578865Z 2025-10-10T00:51:29.2578969Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2579337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2579411Z layer_outputs = layer_module( 2025-10-10T00:51:29.2579652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2579736Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2580046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2580135Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2580436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2580517Z self_outputs = self.self( 2025-10-10T00:51:29.2580817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2580932Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2581302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2581495Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2581507Z 2025-10-10T00:51:29.2581617Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2582031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2582150Z layer_outputs = layer_module( 2025-10-10T00:51:29.2582384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2582477Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2582779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2582859Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2583164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2583237Z self_outputs = self.self( 2025-10-10T00:51:29.2583546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2583657Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2584026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2584218Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2584221Z 2025-10-10T00:51:29.2584309Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2584403Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2584485Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2584574Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2584684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2585063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2585155Z layer_outputs = layer_module( 2025-10-10T00:51:29.2585394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2585484Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2585790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2585874Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2586272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2586358Z self_outputs = self.self( 2025-10-10T00:51:29.2586676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2586795Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2587178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2587332Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2587687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2587806Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2587810Z 2025-10-10T00:51:29.2587896Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2588018Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2588408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2588494Z layer_outputs = layer_module( 2025-10-10T00:51:29.2588779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2588905Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2589217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2589300Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2589609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2589687Z self_outputs = self.self( 2025-10-10T00:51:29.2589987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2590075Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2590079Z 2025-10-10T00:51:29.2590191Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2590590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2590672Z layer_outputs = layer_module( 2025-10-10T00:51:29.2590916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2591002Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2591303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2591393Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2591703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2591786Z self_outputs = self.self( 2025-10-10T00:51:29.2592151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2592244Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2592255Z 2025-10-10T00:51:29.2592366Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2592753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2592838Z layer_outputs = layer_module( 2025-10-10T00:51:29.2593075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2593169Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2593479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2593560Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2593921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2594001Z self_outputs = self.self( 2025-10-10T00:51:29.2594306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2594400Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2594403Z 2025-10-10T00:51:29.2594521Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2594894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2594966Z layer_outputs = layer_module( 2025-10-10T00:51:29.2595197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2595277Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2595639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2595717Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2596012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2596096Z self_outputs = self.self( 2025-10-10T00:51:29.2596398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2596533Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2596916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2597108Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2597324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2597431Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2597435Z 2025-10-10T00:51:29.2597553Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2597929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2598013Z layer_outputs = layer_module( 2025-10-10T00:51:29.2598249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2598340Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2598639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2598719Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2599029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2599105Z self_outputs = self.self( 2025-10-10T00:51:29.2599407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2599530Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2599915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2600068Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2600407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2600517Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2600726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2600835Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2600838Z 2025-10-10T00:51:29.2600948Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2601332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2601419Z layer_outputs = layer_module( 2025-10-10T00:51:29.2601654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2601752Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2602050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2602204Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2602512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2602587Z self_outputs = self.self( 2025-10-10T00:51:29.2602890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2603013Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2603397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2603562Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2603566Z 2025-10-10T00:51:29.2603677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2604067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2604147Z layer_outputs = layer_module( 2025-10-10T00:51:29.2604391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2604478Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2604783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2604865Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2605164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2605249Z self_outputs = self.self( 2025-10-10T00:51:29.2605546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2605684Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2606061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2606232Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2606236Z 2025-10-10T00:51:29.2606346Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2606719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2606803Z layer_outputs = layer_module( 2025-10-10T00:51:29.2607039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2607132Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2607441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2607524Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2607838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2607916Z self_outputs = self.self( 2025-10-10T00:51:29.2608229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2608435Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2608438Z 2025-10-10T00:51:29.2608558Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2608987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2609099Z layer_outputs = layer_module( 2025-10-10T00:51:29.2609343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2609428Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2609732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2609812Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2610114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2610234Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2610533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2610640Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2610644Z 2025-10-10T00:51:29.2610753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2611132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2611211Z layer_outputs = layer_module( 2025-10-10T00:51:29.2611446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2611537Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2611833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2611933Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2612211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2612308Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2612611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2612731Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2613038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2613127Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2613131Z 2025-10-10T00:51:29.2613246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2613629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2613713Z layer_outputs = layer_module( 2025-10-10T00:51:29.2613951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2614038Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2614342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2614432Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2614714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2614797Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2615097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2615221Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2615551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2615713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2615943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2616028Z return self.act(input) 2025-10-10T00:51:29.2616032Z 2025-10-10T00:51:29.2616143Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2616525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2616610Z layer_outputs = layer_module( 2025-10-10T00:51:29.2616845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2616936Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2617236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2617329Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2617612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2617694Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2617999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2618130Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2618435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2618522Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2618525Z 2025-10-10T00:51:29.2618634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2619016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2619098Z layer_outputs = layer_module( 2025-10-10T00:51:29.2619343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2619428Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2619730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2619814Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2620119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2620204Z self_outputs = self.self( 2025-10-10T00:51:29.2620503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2620604Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2620608Z 2025-10-10T00:51:29.2620715Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2621098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2621183Z layer_outputs = layer_module( 2025-10-10T00:51:29.2621419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2621512Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2621818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2621907Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2622247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2622364Z self_outputs = self.self( 2025-10-10T00:51:29.2622669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2622779Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2623152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2623349Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2623353Z 2025-10-10T00:51:29.2623471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2623847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2623930Z layer_outputs = layer_module( 2025-10-10T00:51:29.2624174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2624260Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2624568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2624653Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2624960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2625045Z self_outputs = self.self( 2025-10-10T00:51:29.2625349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2625446Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2625453Z 2025-10-10T00:51:29.2625568Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2626033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2626123Z layer_outputs = layer_module( 2025-10-10T00:51:29.2626373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2626472Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2626785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2626879Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2627191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2627272Z self_outputs = self.self( 2025-10-10T00:51:29.2627601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2627715Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2628104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2628307Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2628312Z 2025-10-10T00:51:29.2628435Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2628825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2628906Z layer_outputs = layer_module( 2025-10-10T00:51:29.2629200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2629325Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2629641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2629726Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2630042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2630119Z self_outputs = self.self( 2025-10-10T00:51:29.2630427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2630550Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2630928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2631136Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2631139Z 2025-10-10T00:51:29.2631255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2631829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2631915Z layer_outputs = layer_module( 2025-10-10T00:51:29.2632160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2632256Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2632567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2632659Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2632975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2633054Z self_outputs = self.self( 2025-10-10T00:51:29.2633368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2633479Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2633862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2634059Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2634063Z 2025-10-10T00:51:29.2634162Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2634255Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2634342Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2634442Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2634554Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2634951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2635032Z layer_outputs = layer_module( 2025-10-10T00:51:29.2635273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2635370Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2635681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2635774Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2636078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2636299Z self_outputs = self.self( 2025-10-10T00:51:29.2636609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2636732Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2637111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2637274Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2637636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2637744Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2637748Z 2025-10-10T00:51:29.2637839Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2637963Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2638357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2638447Z layer_outputs = layer_module( 2025-10-10T00:51:29.2638692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2638788Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2639097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2639175Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2639467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2639539Z self_outputs = self.self( 2025-10-10T00:51:29.2639834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2639918Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2639921Z 2025-10-10T00:51:29.2640032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2640425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2640501Z layer_outputs = layer_module( 2025-10-10T00:51:29.2640745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2640830Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2641145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2641227Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2641531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2641616Z self_outputs = self.self( 2025-10-10T00:51:29.2641914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2642002Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2642006Z 2025-10-10T00:51:29.2642109Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2642471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2642544Z layer_outputs = layer_module( 2025-10-10T00:51:29.2642767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2642854Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2643200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2643286Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2643565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2643635Z self_outputs = self.self( 2025-10-10T00:51:29.2643921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2644007Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2644011Z 2025-10-10T00:51:29.2644122Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2644477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2644561Z layer_outputs = layer_module( 2025-10-10T00:51:29.2644785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2644864Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2645156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2645233Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2645522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2645593Z self_outputs = self.self( 2025-10-10T00:51:29.2645872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2646009Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2646399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2646595Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2646806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2646919Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2646923Z 2025-10-10T00:51:29.2647032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2647418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2647502Z layer_outputs = layer_module( 2025-10-10T00:51:29.2647735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2647833Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2648134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2648223Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2648528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2648603Z self_outputs = self.self( 2025-10-10T00:51:29.2648907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2649034Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2649414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2649679Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2650001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2650103Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2650300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2650410Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2650413Z 2025-10-10T00:51:29.2650524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2650917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2650995Z layer_outputs = layer_module( 2025-10-10T00:51:29.2651232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2651329Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2651629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2651718Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2652023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2652105Z self_outputs = self.self( 2025-10-10T00:51:29.2652405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2652523Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2652886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2653045Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2653049Z 2025-10-10T00:51:29.2653162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2653517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2653595Z layer_outputs = layer_module( 2025-10-10T00:51:29.2653817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2653896Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2654249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2654327Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2654622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2654694Z self_outputs = self.self( 2025-10-10T00:51:29.2654972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2655094Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2655446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2655604Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2655608Z 2025-10-10T00:51:29.2655712Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2656103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2656239Z layer_outputs = layer_module( 2025-10-10T00:51:29.2656463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2656552Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2656838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2656924Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2657208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2657281Z self_outputs = self.self( 2025-10-10T00:51:29.2657573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2657766Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2657771Z 2025-10-10T00:51:29.2657882Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2658239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2658321Z layer_outputs = layer_module( 2025-10-10T00:51:29.2658541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2658621Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2658911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2658989Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2659282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2659401Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2659696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2659789Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2659792Z 2025-10-10T00:51:29.2659901Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2660354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2660429Z layer_outputs = layer_module( 2025-10-10T00:51:29.2660659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2660738Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2661031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2661121Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2661388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2661475Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2661761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2661888Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2662174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2662259Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2662263Z 2025-10-10T00:51:29.2662373Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2662791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2662874Z layer_outputs = layer_module( 2025-10-10T00:51:29.2663093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2663180Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2663484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2663572Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2663858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2663942Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2664256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2664376Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2664676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2664806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2665037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2665122Z return self.act(input) 2025-10-10T00:51:29.2665126Z 2025-10-10T00:51:29.2665235Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2665629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2665708Z layer_outputs = layer_module( 2025-10-10T00:51:29.2666006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2666114Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2666422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2666520Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2666809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2666893Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2667213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2667358Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2667672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2667764Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2667767Z 2025-10-10T00:51:29.2667888Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2668260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2668339Z layer_outputs = layer_module( 2025-10-10T00:51:29.2668585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2668670Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2668980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2669064Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2669444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2669522Z self_outputs = self.self( 2025-10-10T00:51:29.2669819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2669919Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2669923Z 2025-10-10T00:51:29.2670032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2670409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2670487Z layer_outputs = layer_module( 2025-10-10T00:51:29.2670724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2670818Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2671120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2671209Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2671503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2671586Z self_outputs = self.self( 2025-10-10T00:51:29.2671881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2671990Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2672361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2672557Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2672566Z 2025-10-10T00:51:29.2672683Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2673055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2673136Z layer_outputs = layer_module( 2025-10-10T00:51:29.2673371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2673457Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2673761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2673842Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2674145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2674227Z self_outputs = self.self( 2025-10-10T00:51:29.2674526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2674620Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2674624Z 2025-10-10T00:51:29.2674733Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2675112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2675189Z layer_outputs = layer_module( 2025-10-10T00:51:29.2675430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2675516Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2675854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2675979Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2676279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2676363Z self_outputs = self.self( 2025-10-10T00:51:29.2676670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2676781Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2677129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2677312Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2677316Z 2025-10-10T00:51:29.2677428Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2677788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2677868Z layer_outputs = layer_module( 2025-10-10T00:51:29.2678091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2678172Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2678465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2678541Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2678833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2678905Z self_outputs = self.self( 2025-10-10T00:51:29.2679194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2679306Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2679666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2679867Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2679871Z 2025-10-10T00:51:29.2679981Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2680369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2680446Z layer_outputs = layer_module( 2025-10-10T00:51:29.2680690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2680781Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2681080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2681170Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2681468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2681547Z self_outputs = self.self( 2025-10-10T00:51:29.2681831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2681939Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2682305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2682532Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2682567Z 2025-10-10T00:51:29.2682667Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2682755Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2682846Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2682928Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2683038Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2683420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2683499Z layer_outputs = layer_module( 2025-10-10T00:51:29.2683745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2683830Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2684133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2684225Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2684524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2684607Z self_outputs = self.self( 2025-10-10T00:51:29.2684906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2685041Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2685382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2685529Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2685865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2685968Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2685972Z 2025-10-10T00:51:29.2686061Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2686166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2686555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2686632Z layer_outputs = layer_module( 2025-10-10T00:51:29.2686867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2686961Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2687262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2687354Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2687655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2687731Z self_outputs = self.self( 2025-10-10T00:51:29.2688035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2688114Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2688118Z 2025-10-10T00:51:29.2688234Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2688618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2688702Z layer_outputs = layer_module( 2025-10-10T00:51:29.2688938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2689093Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2689399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2689479Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2689780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2689856Z self_outputs = self.self( 2025-10-10T00:51:29.2690153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2690248Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2690252Z 2025-10-10T00:51:29.2690363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2690749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2690829Z layer_outputs = layer_module( 2025-10-10T00:51:29.2691069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2691155Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2691453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2691542Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2691838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2691920Z self_outputs = self.self( 2025-10-10T00:51:29.2692214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2692307Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2692323Z 2025-10-10T00:51:29.2692434Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2692815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2692902Z layer_outputs = layer_module( 2025-10-10T00:51:29.2693134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2693225Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2693521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2693603Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2693907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2693988Z self_outputs = self.self( 2025-10-10T00:51:29.2694289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2694416Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2694806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2694992Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2695202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2695320Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2695324Z 2025-10-10T00:51:29.2695436Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2695860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2695979Z layer_outputs = layer_module( 2025-10-10T00:51:29.2696219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2696313Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2696618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2696707Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2697011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2697093Z self_outputs = self.self( 2025-10-10T00:51:29.2697400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2697528Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2697929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2698076Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2698428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2698528Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2698747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2698852Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2698856Z 2025-10-10T00:51:29.2698966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2699361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2699438Z layer_outputs = layer_module( 2025-10-10T00:51:29.2699686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2699771Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2700078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2700167Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2700470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2700553Z self_outputs = self.self( 2025-10-10T00:51:29.2700859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2700992Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2701377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2701542Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2701546Z 2025-10-10T00:51:29.2701665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2702061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2702146Z layer_outputs = layer_module( 2025-10-10T00:51:29.2702387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2702551Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2702852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2702934Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2703246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2703322Z self_outputs = self.self( 2025-10-10T00:51:29.2703626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2703750Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2704139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2704311Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2704318Z 2025-10-10T00:51:29.2704428Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2704812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2704889Z layer_outputs = layer_module( 2025-10-10T00:51:29.2705135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2705222Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2705524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2705612Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2705985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2706090Z self_outputs = self.self( 2025-10-10T00:51:29.2706396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2706612Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2706616Z 2025-10-10T00:51:29.2706729Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2707115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2707202Z layer_outputs = layer_module( 2025-10-10T00:51:29.2707446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2707541Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2707869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2707951Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2708258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2708382Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2708689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2708781Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2708785Z 2025-10-10T00:51:29.2708902Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2709273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2709434Z layer_outputs = layer_module( 2025-10-10T00:51:29.2709678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2709766Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2710069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2710160Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2710447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2710530Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2710833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2710960Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2711266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2711363Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2711367Z 2025-10-10T00:51:29.2711485Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2711844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2711916Z layer_outputs = layer_module( 2025-10-10T00:51:29.2712150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2712243Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2712544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2712647Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2712929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2713014Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2713325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2713441Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2713745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2713867Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2714104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2714176Z return self.act(input) 2025-10-10T00:51:29.2714180Z 2025-10-10T00:51:29.2714291Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2714649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2714723Z layer_outputs = layer_module( 2025-10-10T00:51:29.2714952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2715033Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2715316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2715410Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2715672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2715757Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2716109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2716243Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2716535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2716622Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2716626Z 2025-10-10T00:51:29.2716744Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2717117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2717202Z layer_outputs = layer_module( 2025-10-10T00:51:29.2717437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2717528Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2717834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2717918Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2718221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2718298Z self_outputs = self.self( 2025-10-10T00:51:29.2718601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2718689Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2718693Z 2025-10-10T00:51:29.2718802Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2719177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2719252Z layer_outputs = layer_module( 2025-10-10T00:51:29.2719481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2719560Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2719845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2719924Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2720205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2720285Z self_outputs = self.self( 2025-10-10T00:51:29.2720565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2720680Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2721024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2721209Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2721222Z 2025-10-10T00:51:29.2721332Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2721712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2721797Z layer_outputs = layer_module( 2025-10-10T00:51:29.2722037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2722131Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2722466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2722591Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2722908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2722985Z self_outputs = self.self( 2025-10-10T00:51:29.2723310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2723397Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2723401Z 2025-10-10T00:51:29.2723515Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2723897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2723974Z layer_outputs = layer_module( 2025-10-10T00:51:29.2724226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2724309Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2724616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2724696Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2724995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2725079Z self_outputs = self.self( 2025-10-10T00:51:29.2725376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2725492Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2725856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2726051Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2726054Z 2025-10-10T00:51:29.2726162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2726550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2726635Z layer_outputs = layer_module( 2025-10-10T00:51:29.2726871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2726962Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2727262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2727351Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2727658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2727735Z self_outputs = self.self( 2025-10-10T00:51:29.2728044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2728153Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2728527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2728721Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2728724Z 2025-10-10T00:51:29.2728852Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2729245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2729350Z layer_outputs = layer_module( 2025-10-10T00:51:29.2729597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2729683Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2729990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2730072Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2730371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2730454Z self_outputs = self.self( 2025-10-10T00:51:29.2730750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2730868Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2731232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2731432Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2731435Z 2025-10-10T00:51:29.2731658Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2731752Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2731845Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2731929Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2732052Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2732423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2732511Z layer_outputs = layer_module( 2025-10-10T00:51:29.2732765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2732851Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2733158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2733240Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2733537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2733623Z self_outputs = self.self( 2025-10-10T00:51:29.2733918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2734047Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2734411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2734576Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2734918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2735025Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2735037Z 2025-10-10T00:51:29.2735122Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2735233Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2735616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2735694Z layer_outputs = layer_module( 2025-10-10T00:51:29.2736008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2736150Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2736447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2736536Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2736834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2736920Z self_outputs = self.self( 2025-10-10T00:51:29.2737218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2737298Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2737311Z 2025-10-10T00:51:29.2737423Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2737832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2737922Z layer_outputs = layer_module( 2025-10-10T00:51:29.2738159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2738253Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2738551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2738633Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2738939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2739016Z self_outputs = self.self( 2025-10-10T00:51:29.2739320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2739413Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2739417Z 2025-10-10T00:51:29.2739527Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2739916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2739993Z layer_outputs = layer_module( 2025-10-10T00:51:29.2740237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2740323Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2740629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2740709Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2741006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2741089Z self_outputs = self.self( 2025-10-10T00:51:29.2741382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2741481Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2741485Z 2025-10-10T00:51:29.2741594Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2741985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2742061Z layer_outputs = layer_module( 2025-10-10T00:51:29.2742296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2742388Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2742724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2742847Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2743143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2743216Z self_outputs = self.self( 2025-10-10T00:51:29.2743518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2743648Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2744029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2744215Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2744434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2744544Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2744548Z 2025-10-10T00:51:29.2744655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2745046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2745124Z layer_outputs = layer_module( 2025-10-10T00:51:29.2745368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2745455Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2745761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2745841Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2746204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2746303Z self_outputs = self.self( 2025-10-10T00:51:29.2746602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2746738Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2747136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2747294Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2747621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2747716Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2747923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2748024Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2748028Z 2025-10-10T00:51:29.2748139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2748491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2748564Z layer_outputs = layer_module( 2025-10-10T00:51:29.2748796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2748878Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2749171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2749249Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2749606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2749691Z self_outputs = self.self( 2025-10-10T00:51:29.2749972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2750098Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2750451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2750612Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2750616Z 2025-10-10T00:51:29.2750721Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2751077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2751160Z layer_outputs = layer_module( 2025-10-10T00:51:29.2751383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2751472Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2751754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2751837Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2752118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2752190Z self_outputs = self.self( 2025-10-10T00:51:29.2752476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2752598Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2752964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2753112Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2753115Z 2025-10-10T00:51:29.2753225Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2753576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2753649Z layer_outputs = layer_module( 2025-10-10T00:51:29.2753879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2753961Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2754262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2754338Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2754611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2754689Z self_outputs = self.self( 2025-10-10T00:51:29.2754959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2755148Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2755152Z 2025-10-10T00:51:29.2755255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2755610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2755743Z layer_outputs = layer_module( 2025-10-10T00:51:29.2755956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2756041Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2756312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2756394Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2756664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2756783Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2757051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2757137Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2757145Z 2025-10-10T00:51:29.2757254Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2757599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2757679Z layer_outputs = layer_module( 2025-10-10T00:51:29.2757900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2757981Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2758266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2758354Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2758619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2758705Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2758992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2759104Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2759384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2759474Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2759477Z 2025-10-10T00:51:29.2759580Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2759924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2759995Z layer_outputs = layer_module( 2025-10-10T00:51:29.2760217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2760298Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2760582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2760677Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2760946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2761034Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2761324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2761439Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2761733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2761938Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2762167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2762241Z return self.act(input) 2025-10-10T00:51:29.2762244Z 2025-10-10T00:51:29.2762357Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2762712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2762787Z layer_outputs = layer_module( 2025-10-10T00:51:29.2763018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2763100Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2763391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2763483Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2763746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2763832Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2764118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2764254Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2764536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2764627Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2764630Z 2025-10-10T00:51:29.2764735Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2765095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2765177Z layer_outputs = layer_module( 2025-10-10T00:51:29.2765395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2765481Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2765756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2765840Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2766119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2766197Z self_outputs = self.self( 2025-10-10T00:51:29.2766502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-10-10T00:51:29.2766595Z query_vectors = self.query(hidden_states) 2025-10-10T00:51:29.2766599Z 2025-10-10T00:51:29.2766716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2767090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2767167Z layer_outputs = layer_module( 2025-10-10T00:51:29.2767410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2767495Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2767799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2767878Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2768208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2768314Z self_outputs = self.self( 2025-10-10T00:51:29.2768584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2768693Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2769029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2769216Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2769219Z 2025-10-10T00:51:29.2769319Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2769677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2769757Z layer_outputs = layer_module( 2025-10-10T00:51:29.2769977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2770065Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2770347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2770434Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2770715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2770787Z self_outputs = self.self( 2025-10-10T00:51:29.2771075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-10-10T00:51:29.2771158Z key_vectors = self.key(hidden_states) 2025-10-10T00:51:29.2771162Z 2025-10-10T00:51:29.2771275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2771656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2771741Z layer_outputs = layer_module( 2025-10-10T00:51:29.2771974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2772059Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2772364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2772444Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2772750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2772825Z self_outputs = self.self( 2025-10-10T00:51:29.2773127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2773244Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2773606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2773810Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2773813Z 2025-10-10T00:51:29.2773934Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2774294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2774366Z layer_outputs = layer_module( 2025-10-10T00:51:29.2774628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2774747Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2775029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2775113Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2775396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2775474Z self_outputs = self.self( 2025-10-10T00:51:29.2775755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2775856Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2776206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2776395Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2776398Z 2025-10-10T00:51:29.2776510Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2776868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2776947Z layer_outputs = layer_module( 2025-10-10T00:51:29.2777181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2777266Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2777575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2777668Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2777960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2778036Z self_outputs = self.self( 2025-10-10T00:51:29.2778318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-10-10T00:51:29.2778427Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2778773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2778963Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-10-10T00:51:29.2778966Z 2025-10-10T00:51:29.2779051Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2779143Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2779223Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2779300Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2779419Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2779778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2779859Z layer_outputs = layer_module( 2025-10-10T00:51:29.2780084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2780169Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2780479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2780560Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2780879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2780956Z self_outputs = self.self( 2025-10-10T00:51:29.2781342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-10-10T00:51:29.2781465Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-10-10T00:51:29.2781826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-10-10T00:51:29.2781988Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-10-10T00:51:29.2782331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-10-10T00:51:29.2782445Z ).where(beginning_mask.bool(), beginning_input) 2025-10-10T00:51:29.2782449Z 2025-10-10T00:51:29.2782533Z cudagraph partition due to non gpu ops 2025-10-10T00:51:29.2782642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2783032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2783109Z layer_outputs = layer_module( 2025-10-10T00:51:29.2783353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2783438Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2783743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2783824Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2784131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2784216Z self_outputs = self.self( 2025-10-10T00:51:29.2784517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-10-10T00:51:29.2784604Z attn_scores += diagonal_mask 2025-10-10T00:51:29.2784608Z 2025-10-10T00:51:29.2784717Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2785109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2785186Z layer_outputs = layer_module( 2025-10-10T00:51:29.2785422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2785514Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2785823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2785916Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2786309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2786392Z self_outputs = self.self( 2025-10-10T00:51:29.2786705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-10-10T00:51:29.2786795Z attn_probs = nn.functional.softmax( 2025-10-10T00:51:29.2786799Z 2025-10-10T00:51:29.2786921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2787317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2787406Z layer_outputs = layer_module( 2025-10-10T00:51:29.2787651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2787739Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2788139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2788226Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2788543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2788621Z self_outputs = self.self( 2025-10-10T00:51:29.2788928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-10-10T00:51:29.2789030Z value_vectors = self.value(hidden_states) 2025-10-10T00:51:29.2789033Z 2025-10-10T00:51:29.2789146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2789603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2789692Z layer_outputs = layer_module( 2025-10-10T00:51:29.2789944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2790030Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2790346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2790437Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2790751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2790839Z self_outputs = self.self( 2025-10-10T00:51:29.2791148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2791280Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2791688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2791885Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-10-10T00:51:29.2792112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2792223Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2792227Z 2025-10-10T00:51:29.2792345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2792734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2792813Z layer_outputs = layer_module( 2025-10-10T00:51:29.2793063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2793154Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2793469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2793554Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2793869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2793946Z self_outputs = self.self( 2025-10-10T00:51:29.2794250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2794387Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2794775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2794970Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-10-10T00:51:29.2795362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-10-10T00:51:29.2795471Z chunked_hidden_states = nn.functional.pad( 2025-10-10T00:51:29.2795694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5406, in pad 2025-10-10T00:51:29.2795801Z return torch._C._nn.pad(input, pad, mode, value) 2025-10-10T00:51:29.2795805Z 2025-10-10T00:51:29.2795922Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2796296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2796381Z layer_outputs = layer_module( 2025-10-10T00:51:29.2796620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2796708Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2797014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2797097Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2797401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2797476Z self_outputs = self.self( 2025-10-10T00:51:29.2797778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2797905Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2798284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2798456Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2798460Z 2025-10-10T00:51:29.2798566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2798924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2799000Z layer_outputs = layer_module( 2025-10-10T00:51:29.2799230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2799310Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2799592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2799678Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2799959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2800064Z self_outputs = self.self( 2025-10-10T00:51:29.2800342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-10-10T00:51:29.2800459Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-10-10T00:51:29.2800819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-10-10T00:51:29.2800972Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-10-10T00:51:29.2800976Z 2025-10-10T00:51:29.2801089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2801439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2801586Z layer_outputs = layer_module( 2025-10-10T00:51:29.2801810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2801892Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2802181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2802265Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2802554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-10-10T00:51:29.2802625Z self_outputs = self.self( 2025-10-10T00:51:29.2802912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-10-10T00:51:29.2803102Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-10-10T00:51:29.2803111Z 2025-10-10T00:51:29.2803215Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2803580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2803658Z layer_outputs = layer_module( 2025-10-10T00:51:29.2803900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2803984Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2804278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-10-10T00:51:29.2804368Z self_attn_outputs = self.attention( 2025-10-10T00:51:29.2804664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-10-10T00:51:29.2804800Z attn_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:51:29.2805099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-10-10T00:51:29.2805198Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2805202Z 2025-10-10T00:51:29.2805311Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2805676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2805757Z layer_outputs = layer_module( 2025-10-10T00:51:29.2805977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2806065Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2806351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2806448Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2806720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2806803Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2807114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2807234Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2807538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-10-10T00:51:29.2807629Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2807632Z 2025-10-10T00:51:29.2807741Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2808195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2808275Z layer_outputs = layer_module( 2025-10-10T00:51:29.2808521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2808602Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2808886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2808973Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2809247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2809338Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2809641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-10-10T00:51:29.2809768Z intermediate_output = self.intermediate(attn_output) 2025-10-10T00:51:29.2810065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-10-10T00:51:29.2810184Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:51:29.2810421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:51:29.2810498Z return self.act(input) 2025-10-10T00:51:29.2810502Z 2025-10-10T00:51:29.2810617Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:51:29.2810988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-10-10T00:51:29.2811072Z layer_outputs = layer_module( 2025-10-10T00:51:29.2811307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:51:29.2811394Z return super().__call__(*args, **kwargs) 2025-10-10T00:51:29.2811698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-10-10T00:51:29.2811788Z layer_output = apply_chunking_to_forward( 2025-10-10T00:51:29.2812069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:51:29.2812150Z return forward_fn(*input_tensors) 2025-10-10T00:51:29.2812455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-10-10T00:51:29.2812588Z layer_output = self.output(intermediate_output, attn_output) 2025-10-10T00:51:29.2812889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-10-10T00:51:29.2812987Z hidden_states = self.dense(hidden_states) 2025-10-10T00:51:29.2812991Z 2025-10-10T00:52:39.8066808Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:52:39.8070091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-10-10T00:52:39.8070730Z prediction_scores = self.lm_head(sequence_output) 2025-10-10T00:52:39.8071238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-10-10T00:52:39.8071701Z x = self.dense(features) 2025-10-10T00:52:39.8071841Z 2025-10-10T00:52:39.8071971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:52:39.8072961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-10-10T00:52:39.8073650Z prediction_scores = self.lm_head(sequence_output) 2025-10-10T00:52:39.8074132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-10-10T00:52:39.8074577Z x = self.decoder(x) 2025-10-10T00:52:39.8074697Z 2025-10-10T00:52:39.8074819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:52:39.8075370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-10-10T00:52:39.8076017Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:52:39.8076289Z 2025-10-10T00:52:41.1718685Z Compilation time (from dynamo_timed): 104.174806653 2025-10-10T00:52:41.1945287Z pass 2025-10-10T00:52:41.1946182Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:52:41.1947156Z TIMING: gc:0.00473 entire_frame_compile:104.17481 _recursive_pre_grad_passes:0.02014 _recursive_joint_graph_passes:1.03071 _recursive_post_grad_passes:1.67448 async_compile.wait:2.86749 code_gen:81.79422 inductor_compile:89.26657 backend_compile:98.94238 total_wall_time:104.17481 2025-10-10T00:52:41.1953048Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:39126 | FakeTensor.__torch_dispatch__:16210 | ProxyTorchDispatchMode.__torch_dispatch__:8711 2025-10-10T00:52:41.1958422Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-10-10T00:52:45.1933585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:52:45.1934557Z import pynvml # type: ignore[import] 2025-10-10T00:52:48.6682100Z 2025-10-10T00:52:51.3904366Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:52:51.3904681Z loading model: 0it [00:02, ?it/s] 2025-10-10T00:52:51.3926835Z cpu eval BartForCausalLM 2025-10-10T00:52:53.0685097Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:52:53.7078887Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:52:54.3606437Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:02.0455639Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0456025Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0456271Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0456504Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0456734Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0456959Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0460029Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0460781Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0461105Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0461359Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0461606Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0461837Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0462117Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0462637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0463144Z res = mod(**inputs) 2025-10-10T00:53:02.0463629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0464095Z outputs = self.model.decoder( 2025-10-10T00:53:02.0464549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0465573Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0466146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0466581Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0467035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0467470Z return func(*args, **kwargs) 2025-10-10T00:53:02.0467910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0468400Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0468860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0469293Z return func(*args, **kwargs) 2025-10-10T00:53:02.0469717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0470262Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0470496Z 2025-10-10T00:53:02.0470628Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0471032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0471405Z res = mod(**inputs) 2025-10-10T00:53:02.0471822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0472278Z outputs = self.model.decoder( 2025-10-10T00:53:02.0472705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0473213Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0473618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0474043Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0474483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0474899Z return func(*args, **kwargs) 2025-10-10T00:53:02.0475318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0475853Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0476341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0476770Z return func(*args, **kwargs) 2025-10-10T00:53:02.0477179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0477624Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0477787Z 2025-10-10T00:53:02.0477915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0478321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0478684Z res = mod(**inputs) 2025-10-10T00:53:02.0479110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0479547Z outputs = self.model.decoder( 2025-10-10T00:53:02.0479977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0480408Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0480830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0481245Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0481744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0482216Z return func(*args, **kwargs) 2025-10-10T00:53:02.0482615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0483064Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0483501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0483913Z return func(*args, **kwargs) 2025-10-10T00:53:02.0484318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0484748Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0484914Z 2025-10-10T00:53:02.0485005Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0485269Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0485670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0486020Z res = mod(**inputs) 2025-10-10T00:53:02.0486418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0486851Z outputs = self.model.decoder( 2025-10-10T00:53:02.0487261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0487682Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0488062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0488462Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0488877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0489290Z return func(*args, **kwargs) 2025-10-10T00:53:02.0489718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0490165Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0490612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0491019Z return func(*args, **kwargs) 2025-10-10T00:53:02.0491426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0491891Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0492410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0492972Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0493181Z 2025-10-10T00:53:02.0493317Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0493747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0494114Z res = mod(**inputs) 2025-10-10T00:53:02.0494512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0494943Z outputs = self.model.decoder( 2025-10-10T00:53:02.0495374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0495807Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0496188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0496594Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0497058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0497504Z return func(*args, **kwargs) 2025-10-10T00:53:02.0497905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0498362Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0498815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0499217Z return func(*args, **kwargs) 2025-10-10T00:53:02.0499622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0500053Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0500212Z 2025-10-10T00:53:02.0500330Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0500728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0501104Z res = mod(**inputs) 2025-10-10T00:53:02.0501505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0501924Z outputs = self.model.decoder( 2025-10-10T00:53:02.0502344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0502772Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0503170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0503568Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0503995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0504403Z return func(*args, **kwargs) 2025-10-10T00:53:02.0504815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0505298Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0505490Z 2025-10-10T00:53:02.0505606Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0506103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0506485Z res = mod(**inputs) 2025-10-10T00:53:02.0506889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0507370Z outputs = self.model.decoder( 2025-10-10T00:53:02.0507848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0508284Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0508682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0509101Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0509533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0509962Z return func(*args, **kwargs) 2025-10-10T00:53:02.0510386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0510877Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0511318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0511706Z return self.act(input) 2025-10-10T00:53:02.0511843Z 2025-10-10T00:53:02.0511964Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0512371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0512734Z res = mod(**inputs) 2025-10-10T00:53:02.0513230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0513656Z outputs = self.model.decoder( 2025-10-10T00:53:02.0514087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0514502Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0514882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0515266Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0515679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0516079Z return func(*args, **kwargs) 2025-10-10T00:53:02.0516480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0516910Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0517063Z 2025-10-10T00:53:02.0517177Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0517565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0517915Z res = mod(**inputs) 2025-10-10T00:53:02.0518299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0518709Z outputs = self.model.decoder( 2025-10-10T00:53:02.0519123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0519539Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0519920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0520311Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0520725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0521127Z return func(*args, **kwargs) 2025-10-10T00:53:02.0521525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0521964Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0522391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0522786Z return func(*args, **kwargs) 2025-10-10T00:53:02.0523181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0523680Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0523899Z 2025-10-10T00:53:02.0524021Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0524414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0524759Z res = mod(**inputs) 2025-10-10T00:53:02.0525146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0525562Z outputs = self.model.decoder( 2025-10-10T00:53:02.0525965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0526372Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0526752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0527146Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0527564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0528051Z return func(*args, **kwargs) 2025-10-10T00:53:02.0528446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0528892Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0529321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0529729Z return func(*args, **kwargs) 2025-10-10T00:53:02.0530120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0530541Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0530697Z 2025-10-10T00:53:02.0530812Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0531203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0531821Z res = mod(**inputs) 2025-10-10T00:53:02.0532225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0532662Z outputs = self.model.decoder( 2025-10-10T00:53:02.0533086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0533553Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0533941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0534342Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0534768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0535208Z return func(*args, **kwargs) 2025-10-10T00:53:02.0535632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0536093Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0536547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0536976Z return func(*args, **kwargs) 2025-10-10T00:53:02.0537399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0537852Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0538013Z 2025-10-10T00:53:02.0538104Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0538379Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0538783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0539139Z res = mod(**inputs) 2025-10-10T00:53:02.0539522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0539959Z outputs = self.model.decoder( 2025-10-10T00:53:02.0540377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0540814Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0541211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0541616Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0542047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0542474Z return func(*args, **kwargs) 2025-10-10T00:53:02.0542893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0543351Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0543882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0544998Z return func(*args, **kwargs) 2025-10-10T00:53:02.0545415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0545872Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0546446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0546999Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0547214Z 2025-10-10T00:53:02.0547336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0547747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0548110Z res = mod(**inputs) 2025-10-10T00:53:02.0548518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0548926Z outputs = self.model.decoder( 2025-10-10T00:53:02.0549314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0549719Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0550103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0550492Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0550904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0551306Z return func(*args, **kwargs) 2025-10-10T00:53:02.0551727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0552164Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0552591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0552996Z return func(*args, **kwargs) 2025-10-10T00:53:02.0553394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0553815Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0553964Z 2025-10-10T00:53:02.0554080Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0554467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0554816Z res = mod(**inputs) 2025-10-10T00:53:02.0555204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0555620Z outputs = self.model.decoder( 2025-10-10T00:53:02.0556020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0556437Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0556821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0557217Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0557621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0558032Z return func(*args, **kwargs) 2025-10-10T00:53:02.0558428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0558886Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0559074Z 2025-10-10T00:53:02.0559194Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0559634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0560023Z res = mod(**inputs) 2025-10-10T00:53:02.0560411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0560830Z outputs = self.model.decoder( 2025-10-10T00:53:02.0561232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0561638Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0562019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0562412Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0562823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0563227Z return func(*args, **kwargs) 2025-10-10T00:53:02.0563631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0564092Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0564513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0564898Z return self.act(input) 2025-10-10T00:53:02.0565017Z 2025-10-10T00:53:02.0565132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0565533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0565890Z res = mod(**inputs) 2025-10-10T00:53:02.0566277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0566697Z outputs = self.model.decoder( 2025-10-10T00:53:02.0567103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0567521Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0567902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0568308Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0568724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0569135Z return func(*args, **kwargs) 2025-10-10T00:53:02.0569537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0569986Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0570135Z 2025-10-10T00:53:02.0570254Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0570636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0571013Z res = mod(**inputs) 2025-10-10T00:53:02.0571409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0571837Z outputs = self.model.decoder( 2025-10-10T00:53:02.0572241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0572668Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0573057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0573469Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0573890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0574307Z return func(*args, **kwargs) 2025-10-10T00:53:02.0574781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0575271Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0575714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0576133Z return func(*args, **kwargs) 2025-10-10T00:53:02.0576533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0577047Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0577279Z 2025-10-10T00:53:02.0577393Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0577796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0578174Z res = mod(**inputs) 2025-10-10T00:53:02.0578566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0579000Z outputs = self.model.decoder( 2025-10-10T00:53:02.0579421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0579847Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0580229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0580637Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0581058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0581482Z return func(*args, **kwargs) 2025-10-10T00:53:02.0581891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0582332Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0582781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0583188Z return func(*args, **kwargs) 2025-10-10T00:53:02.0583594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0584027Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0584179Z 2025-10-10T00:53:02.0584296Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0584694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0585050Z res = mod(**inputs) 2025-10-10T00:53:02.0585445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0585866Z outputs = self.model.decoder( 2025-10-10T00:53:02.0586397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0586835Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0587236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0587633Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0588037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0588441Z return func(*args, **kwargs) 2025-10-10T00:53:02.0588846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0589292Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0589720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0590125Z return func(*args, **kwargs) 2025-10-10T00:53:02.0590609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0591039Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0591195Z 2025-10-10T00:53:02.0591292Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0591797Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0592185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0592531Z res = mod(**inputs) 2025-10-10T00:53:02.0592919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0593340Z outputs = self.model.decoder( 2025-10-10T00:53:02.0593740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0594150Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0594518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0594886Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0595271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0595673Z return func(*args, **kwargs) 2025-10-10T00:53:02.0596066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0596499Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0596926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0597309Z return func(*args, **kwargs) 2025-10-10T00:53:02.0597683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0598105Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0598563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0599059Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0599246Z 2025-10-10T00:53:02.0599355Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0599723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0600050Z res = mod(**inputs) 2025-10-10T00:53:02.0600415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0600810Z outputs = self.model.decoder( 2025-10-10T00:53:02.0601186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0601583Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0601945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0602331Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0602737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0603138Z return func(*args, **kwargs) 2025-10-10T00:53:02.0603539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0603976Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0604400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0604800Z return func(*args, **kwargs) 2025-10-10T00:53:02.0605294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0605761Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0605916Z 2025-10-10T00:53:02.0606040Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0606451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0606794Z res = mod(**inputs) 2025-10-10T00:53:02.0607178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0607594Z outputs = self.model.decoder( 2025-10-10T00:53:02.0608003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0608414Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0608796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0609196Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0609607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0610009Z return func(*args, **kwargs) 2025-10-10T00:53:02.0610398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0610861Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0611060Z 2025-10-10T00:53:02.0611177Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0611575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0611933Z res = mod(**inputs) 2025-10-10T00:53:02.0612343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0612792Z outputs = self.model.decoder( 2025-10-10T00:53:02.0613200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0613612Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0613983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0614368Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0614775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0615174Z return func(*args, **kwargs) 2025-10-10T00:53:02.0615584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0616047Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0616481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0616868Z return self.act(input) 2025-10-10T00:53:02.0616993Z 2025-10-10T00:53:02.0617124Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0617504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0617854Z res = mod(**inputs) 2025-10-10T00:53:02.0618235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0618652Z outputs = self.model.decoder( 2025-10-10T00:53:02.0619064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0619498Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0619883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0620330Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0620788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0621199Z return func(*args, **kwargs) 2025-10-10T00:53:02.0621604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0622036Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0622190Z 2025-10-10T00:53:02.0622315Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0622706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0623058Z res = mod(**inputs) 2025-10-10T00:53:02.0623457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0623886Z outputs = self.model.decoder( 2025-10-10T00:53:02.0624306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0624752Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0625132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0625536Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0626041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0626487Z return func(*args, **kwargs) 2025-10-10T00:53:02.0626890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0627346Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0627813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0628239Z return func(*args, **kwargs) 2025-10-10T00:53:02.0628638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0629144Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0629388Z 2025-10-10T00:53:02.0629507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0629913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0630278Z res = mod(**inputs) 2025-10-10T00:53:02.0630677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0631121Z outputs = self.model.decoder( 2025-10-10T00:53:02.0631698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0632192Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0632593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0632997Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0633448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0633867Z return func(*args, **kwargs) 2025-10-10T00:53:02.0634278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0634727Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0635169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0635591Z return func(*args, **kwargs) 2025-10-10T00:53:02.0636099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0636590Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0636745Z 2025-10-10T00:53:02.0636872Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0637263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0637625Z res = mod(**inputs) 2025-10-10T00:53:02.0638021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0638474Z outputs = self.model.decoder( 2025-10-10T00:53:02.0638876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0639314Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0639676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0640204Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0640603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0640981Z return func(*args, **kwargs) 2025-10-10T00:53:02.0641371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0641797Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0642214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0642629Z return func(*args, **kwargs) 2025-10-10T00:53:02.0643033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0643474Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0643641Z 2025-10-10T00:53:02.0643729Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0644000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0644386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0644755Z res = mod(**inputs) 2025-10-10T00:53:02.0645164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0645599Z outputs = self.model.decoder( 2025-10-10T00:53:02.0646012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0646437Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0646803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0647183Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0647580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0647988Z return func(*args, **kwargs) 2025-10-10T00:53:02.0648397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0648845Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0649283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0649690Z return func(*args, **kwargs) 2025-10-10T00:53:02.0650090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0650543Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0651033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0651626Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0651864Z 2025-10-10T00:53:02.0651988Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0652382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0652731Z res = mod(**inputs) 2025-10-10T00:53:02.0653123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0653546Z outputs = self.model.decoder( 2025-10-10T00:53:02.0653957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0654373Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0654765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0655172Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0655603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0656005Z return func(*args, **kwargs) 2025-10-10T00:53:02.0656417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0656889Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0657318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0657718Z return func(*args, **kwargs) 2025-10-10T00:53:02.0658113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0658556Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0658719Z 2025-10-10T00:53:02.0658835Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0659313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0659690Z res = mod(**inputs) 2025-10-10T00:53:02.0660090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0660517Z outputs = self.model.decoder( 2025-10-10T00:53:02.0660943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0661370Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0661763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0662167Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0662590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0663005Z return func(*args, **kwargs) 2025-10-10T00:53:02.0663426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0663982Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0664183Z 2025-10-10T00:53:02.0664300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0664702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0665073Z res = mod(**inputs) 2025-10-10T00:53:02.0665472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0665911Z outputs = self.model.decoder( 2025-10-10T00:53:02.0666411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0666868Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0667317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0667768Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0668187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0668588Z return func(*args, **kwargs) 2025-10-10T00:53:02.0668987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0669442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0669864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0670232Z return self.act(input) 2025-10-10T00:53:02.0670349Z 2025-10-10T00:53:02.0670463Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0670834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0671158Z res = mod(**inputs) 2025-10-10T00:53:02.0671516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0671887Z outputs = self.model.decoder( 2025-10-10T00:53:02.0672256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0672632Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0672994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0673367Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0673758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0674131Z return func(*args, **kwargs) 2025-10-10T00:53:02.0674530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0674962Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0675115Z 2025-10-10T00:53:02.0675237Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0675624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0675972Z res = mod(**inputs) 2025-10-10T00:53:02.0676361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0676751Z outputs = self.model.decoder( 2025-10-10T00:53:02.0677142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0677523Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0677866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0678236Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0678625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0679005Z return func(*args, **kwargs) 2025-10-10T00:53:02.0679374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0679795Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0680207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0680601Z return func(*args, **kwargs) 2025-10-10T00:53:02.0680977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0681439Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0681705Z 2025-10-10T00:53:02.0681861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0682252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0682602Z res = mod(**inputs) 2025-10-10T00:53:02.0682988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0683395Z outputs = self.model.decoder( 2025-10-10T00:53:02.0683798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0684224Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0684606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0684997Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0685415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0685822Z return func(*args, **kwargs) 2025-10-10T00:53:02.0686223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0686664Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0687094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0687515Z return func(*args, **kwargs) 2025-10-10T00:53:02.0687915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0688347Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0688498Z 2025-10-10T00:53:02.0688624Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0689022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0689386Z res = mod(**inputs) 2025-10-10T00:53:02.0689781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0690201Z outputs = self.model.decoder( 2025-10-10T00:53:02.0690607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0691027Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0691411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0691802Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0692222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0692623Z return func(*args, **kwargs) 2025-10-10T00:53:02.0693022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0693466Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0693899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0694347Z return func(*args, **kwargs) 2025-10-10T00:53:02.0694757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0695197Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0695350Z 2025-10-10T00:53:02.0695437Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0695701Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0696088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0696440Z res = mod(**inputs) 2025-10-10T00:53:02.0696888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0697332Z outputs = self.model.decoder( 2025-10-10T00:53:02.0697743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0698159Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0698539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0698932Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0699342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0699742Z return func(*args, **kwargs) 2025-10-10T00:53:02.0700158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0700615Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0701054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0701476Z return func(*args, **kwargs) 2025-10-10T00:53:02.0701883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0702344Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0702843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0703376Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0703590Z 2025-10-10T00:53:02.0703705Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0704106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0704477Z res = mod(**inputs) 2025-10-10T00:53:02.0704868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0705303Z outputs = self.model.decoder( 2025-10-10T00:53:02.0705722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0706257Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0706662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0707069Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0707506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0707930Z return func(*args, **kwargs) 2025-10-10T00:53:02.0708352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0708819Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0709263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0709683Z return func(*args, **kwargs) 2025-10-10T00:53:02.0710096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0710541Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0710697Z 2025-10-10T00:53:02.0710814Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0711215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0711575Z res = mod(**inputs) 2025-10-10T00:53:02.0711976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0712499Z outputs = self.model.decoder( 2025-10-10T00:53:02.0712922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0713355Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0713754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0714173Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0714607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0715024Z return func(*args, **kwargs) 2025-10-10T00:53:02.0715447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0715935Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0716134Z 2025-10-10T00:53:02.0716266Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0716668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0717036Z res = mod(**inputs) 2025-10-10T00:53:02.0717440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0717883Z outputs = self.model.decoder( 2025-10-10T00:53:02.0718298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0718720Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0719112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0719526Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0719956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0720385Z return func(*args, **kwargs) 2025-10-10T00:53:02.0720792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0721268Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0721709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0722113Z return self.act(input) 2025-10-10T00:53:02.0722240Z 2025-10-10T00:53:02.0722363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0722770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0723138Z res = mod(**inputs) 2025-10-10T00:53:02.0723549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0723972Z outputs = self.model.decoder( 2025-10-10T00:53:02.0724388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0724821Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0725225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0725650Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0726077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0726511Z return func(*args, **kwargs) 2025-10-10T00:53:02.0726931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0727394Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0727559Z 2025-10-10T00:53:02.0727688Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0728195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0728565Z res = mod(**inputs) 2025-10-10T00:53:02.0728977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0729401Z outputs = self.model.decoder( 2025-10-10T00:53:02.0729809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0730213Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0730589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0730992Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0731409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0732022Z return func(*args, **kwargs) 2025-10-10T00:53:02.0732448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0732892Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0733379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0733760Z return func(*args, **kwargs) 2025-10-10T00:53:02.0734139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0734635Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0734871Z 2025-10-10T00:53:02.0734985Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0735386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0735720Z res = mod(**inputs) 2025-10-10T00:53:02.0736087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0736480Z outputs = self.model.decoder( 2025-10-10T00:53:02.0736865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0737262Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0737613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0737987Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0738377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0738756Z return func(*args, **kwargs) 2025-10-10T00:53:02.0739153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0739594Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0740026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0740427Z return func(*args, **kwargs) 2025-10-10T00:53:02.0740824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0741246Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0741392Z 2025-10-10T00:53:02.0741505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0741890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0742239Z res = mod(**inputs) 2025-10-10T00:53:02.0742623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0743035Z outputs = self.model.decoder( 2025-10-10T00:53:02.0743597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0744013Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0744396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0744810Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0745239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0745649Z return func(*args, **kwargs) 2025-10-10T00:53:02.0746115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0746578Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0747030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0747430Z return func(*args, **kwargs) 2025-10-10T00:53:02.0747821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0748229Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0748376Z 2025-10-10T00:53:02.0748470Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0748722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0749095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0749428Z res = mod(**inputs) 2025-10-10T00:53:02.0749793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0750184Z outputs = self.model.decoder( 2025-10-10T00:53:02.0750569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0750953Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0751308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0751706Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0752114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0752493Z return func(*args, **kwargs) 2025-10-10T00:53:02.0752867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0753278Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0753685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0754060Z return func(*args, **kwargs) 2025-10-10T00:53:02.0754433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0754846Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0755298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0755793Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0755982Z 2025-10-10T00:53:02.0756088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0756460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0756777Z res = mod(**inputs) 2025-10-10T00:53:02.0757132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0757507Z outputs = self.model.decoder( 2025-10-10T00:53:02.0757929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0758355Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0758715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0759088Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0759478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0759861Z return func(*args, **kwargs) 2025-10-10T00:53:02.0760242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0760666Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0761075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0761449Z return func(*args, **kwargs) 2025-10-10T00:53:02.0761839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0762247Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0762390Z 2025-10-10T00:53:02.0762507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0762871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0763201Z res = mod(**inputs) 2025-10-10T00:53:02.0763568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0763968Z outputs = self.model.decoder( 2025-10-10T00:53:02.0764342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0764717Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0765070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0765438Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0765828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0766202Z return func(*args, **kwargs) 2025-10-10T00:53:02.0766559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0766985Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0767160Z 2025-10-10T00:53:02.0767266Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0767623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0767936Z res = mod(**inputs) 2025-10-10T00:53:02.0768294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0768685Z outputs = self.model.decoder( 2025-10-10T00:53:02.0769072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0769466Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0769823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0770218Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0770625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0771010Z return func(*args, **kwargs) 2025-10-10T00:53:02.0771389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0771834Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0772335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0772691Z return self.act(input) 2025-10-10T00:53:02.0772805Z 2025-10-10T00:53:02.0772918Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0773283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0773614Z res = mod(**inputs) 2025-10-10T00:53:02.0773977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0774377Z outputs = self.model.decoder( 2025-10-10T00:53:02.0774776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0775159Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0775528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0775893Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0776271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0776672Z return func(*args, **kwargs) 2025-10-10T00:53:02.0777049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0777445Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0777587Z 2025-10-10T00:53:02.0777702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0778069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0778448Z res = mod(**inputs) 2025-10-10T00:53:02.0778800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0779197Z outputs = self.model.decoder( 2025-10-10T00:53:02.0779583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0779976Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0780340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0780734Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0781136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0781515Z return func(*args, **kwargs) 2025-10-10T00:53:02.0781885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0782326Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0782759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0783160Z return func(*args, **kwargs) 2025-10-10T00:53:02.0783556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0784042Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0784271Z 2025-10-10T00:53:02.0784384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0784775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0785121Z res = mod(**inputs) 2025-10-10T00:53:02.0785507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0785915Z outputs = self.model.decoder( 2025-10-10T00:53:02.0786439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0786904Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0787302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0787689Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0788102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0788501Z return func(*args, **kwargs) 2025-10-10T00:53:02.0788903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0789343Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0789762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0790162Z return func(*args, **kwargs) 2025-10-10T00:53:02.0790566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0790986Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0791135Z 2025-10-10T00:53:02.0791255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0791633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0791978Z res = mod(**inputs) 2025-10-10T00:53:02.0792358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0792769Z outputs = self.model.decoder( 2025-10-10T00:53:02.0793166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0793581Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0793961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0794360Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0794768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0795159Z return func(*args, **kwargs) 2025-10-10T00:53:02.0795557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0795993Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0796419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0796823Z return func(*args, **kwargs) 2025-10-10T00:53:02.0797201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0797607Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0797768Z 2025-10-10T00:53:02.0797855Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0798106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0798462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0798793Z res = mod(**inputs) 2025-10-10T00:53:02.0799158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0799573Z outputs = self.model.decoder( 2025-10-10T00:53:02.0799982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0800389Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0800769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0801162Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0801661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0802035Z return func(*args, **kwargs) 2025-10-10T00:53:02.0802410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0802847Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0803279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0803679Z return func(*args, **kwargs) 2025-10-10T00:53:02.0804066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0804517Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0804975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0805470Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0805659Z 2025-10-10T00:53:02.0805778Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0806158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0806508Z res = mod(**inputs) 2025-10-10T00:53:02.0806896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0807312Z outputs = self.model.decoder( 2025-10-10T00:53:02.0807711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0808121Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0808499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0808900Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0809283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0809653Z return func(*args, **kwargs) 2025-10-10T00:53:02.0810033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0810469Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0810894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0811291Z return func(*args, **kwargs) 2025-10-10T00:53:02.0811677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0812099Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0812249Z 2025-10-10T00:53:02.0812362Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0812730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0813049Z res = mod(**inputs) 2025-10-10T00:53:02.0813415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0813814Z outputs = self.model.decoder( 2025-10-10T00:53:02.0814219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0814635Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0815005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0815396Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0815804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0816335Z return func(*args, **kwargs) 2025-10-10T00:53:02.0816731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0817188Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0817384Z 2025-10-10T00:53:02.0817498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0817883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0818288Z res = mod(**inputs) 2025-10-10T00:53:02.0818668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0819081Z outputs = self.model.decoder( 2025-10-10T00:53:02.0819486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0819905Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0820283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0820696Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0821105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0821505Z return func(*args, **kwargs) 2025-10-10T00:53:02.0821899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0822356Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0822769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0823143Z return self.act(input) 2025-10-10T00:53:02.0823271Z 2025-10-10T00:53:02.0823385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0823775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0824116Z res = mod(**inputs) 2025-10-10T00:53:02.0824499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0824910Z outputs = self.model.decoder( 2025-10-10T00:53:02.0825312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0825724Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0826199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0826621Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0827046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0827472Z return func(*args, **kwargs) 2025-10-10T00:53:02.0827869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0828294Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0828454Z 2025-10-10T00:53:02.0828566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0828954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0829307Z res = mod(**inputs) 2025-10-10T00:53:02.0829690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0830107Z outputs = self.model.decoder( 2025-10-10T00:53:02.0830514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0830927Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0831411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0831983Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0832405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0832812Z return func(*args, **kwargs) 2025-10-10T00:53:02.0833213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0833653Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0834091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0834500Z return func(*args, **kwargs) 2025-10-10T00:53:02.0834903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0835408Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0835630Z 2025-10-10T00:53:02.0835744Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0836131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0836453Z res = mod(**inputs) 2025-10-10T00:53:02.0836812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0837198Z outputs = self.model.decoder( 2025-10-10T00:53:02.0837574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0837962Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0838322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0838706Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0839089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0839472Z return func(*args, **kwargs) 2025-10-10T00:53:02.0839849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0840267Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0840676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0841044Z return func(*args, **kwargs) 2025-10-10T00:53:02.0841421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0841830Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0841978Z 2025-10-10T00:53:02.0842101Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0842499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0842844Z res = mod(**inputs) 2025-10-10T00:53:02.0843232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0843657Z outputs = self.model.decoder( 2025-10-10T00:53:02.0844031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0844403Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0844756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0845119Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0845497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0845986Z return func(*args, **kwargs) 2025-10-10T00:53:02.0846350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0846760Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0847152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0847526Z return func(*args, **kwargs) 2025-10-10T00:53:02.0847902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0848296Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0848448Z 2025-10-10T00:53:02.0848534Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0848789Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0849171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0849525Z res = mod(**inputs) 2025-10-10T00:53:02.0849888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0850283Z outputs = self.model.decoder( 2025-10-10T00:53:02.0850679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0851093Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0851460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0851852Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0852258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0852659Z return func(*args, **kwargs) 2025-10-10T00:53:02.0853047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0853488Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0853911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0854312Z return func(*args, **kwargs) 2025-10-10T00:53:02.0854707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0855136Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0855616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0856138Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0856333Z 2025-10-10T00:53:02.0856457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0856847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0857189Z res = mod(**inputs) 2025-10-10T00:53:02.0857575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0857987Z outputs = self.model.decoder( 2025-10-10T00:53:02.0858392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0858807Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0859177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0859574Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0860006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0860428Z return func(*args, **kwargs) 2025-10-10T00:53:02.0860918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0861370Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0861821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0862245Z return func(*args, **kwargs) 2025-10-10T00:53:02.0862651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0863077Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0863238Z 2025-10-10T00:53:02.0863354Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0863753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0864121Z res = mod(**inputs) 2025-10-10T00:53:02.0864512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0864943Z outputs = self.model.decoder( 2025-10-10T00:53:02.0865360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0865792Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0866247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0866651Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0867075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0867494Z return func(*args, **kwargs) 2025-10-10T00:53:02.0867914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0868375Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0868571Z 2025-10-10T00:53:02.0868687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0869075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0869424Z res = mod(**inputs) 2025-10-10T00:53:02.0869812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0870225Z outputs = self.model.decoder( 2025-10-10T00:53:02.0870632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0871046Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0871429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0871821Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0872226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0872631Z return func(*args, **kwargs) 2025-10-10T00:53:02.0873030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0873497Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0873930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0874314Z return self.act(input) 2025-10-10T00:53:02.0874446Z 2025-10-10T00:53:02.0874563Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0874959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0875315Z res = mod(**inputs) 2025-10-10T00:53:02.0875702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0876207Z outputs = self.model.decoder( 2025-10-10T00:53:02.0876630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0877057Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0877448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0877846Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0878267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0878679Z return func(*args, **kwargs) 2025-10-10T00:53:02.0879092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0879524Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0879678Z 2025-10-10T00:53:02.0879800Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0880195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0880552Z res = mod(**inputs) 2025-10-10T00:53:02.0880945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0881362Z outputs = self.model.decoder( 2025-10-10T00:53:02.0881782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0882206Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0882595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0882996Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0883408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0883829Z return func(*args, **kwargs) 2025-10-10T00:53:02.0884241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0884690Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0885122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0885531Z return func(*args, **kwargs) 2025-10-10T00:53:02.0885938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0886453Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0886673Z 2025-10-10T00:53:02.0886793Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0887170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0887519Z res = mod(**inputs) 2025-10-10T00:53:02.0887903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0888317Z outputs = self.model.decoder( 2025-10-10T00:53:02.0888722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0889131Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0889514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0889906Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0890316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0890705Z return func(*args, **kwargs) 2025-10-10T00:53:02.0891137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0891618Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0892047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0892451Z return func(*args, **kwargs) 2025-10-10T00:53:02.0892845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0893269Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0893425Z 2025-10-10T00:53:02.0893541Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0893930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0894277Z res = mod(**inputs) 2025-10-10T00:53:02.0894657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0895077Z outputs = self.model.decoder( 2025-10-10T00:53:02.0895484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0895901Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0896275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0896675Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0897091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0897495Z return func(*args, **kwargs) 2025-10-10T00:53:02.0897895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0898329Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0898763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0899170Z return func(*args, **kwargs) 2025-10-10T00:53:02.0899569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0899996Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0900151Z 2025-10-10T00:53:02.0900241Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0900508Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0900896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0901242Z res = mod(**inputs) 2025-10-10T00:53:02.0901625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0902065Z outputs = self.model.decoder( 2025-10-10T00:53:02.0902501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0902940Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0903367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0903758Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0904187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0904635Z return func(*args, **kwargs) 2025-10-10T00:53:02.0905046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0905496Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0906008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0906535Z return func(*args, **kwargs) 2025-10-10T00:53:02.0906952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0907410Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0907905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0908450Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0908665Z 2025-10-10T00:53:02.0908783Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0909187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0909552Z res = mod(**inputs) 2025-10-10T00:53:02.0909945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0910379Z outputs = self.model.decoder( 2025-10-10T00:53:02.0910809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0911245Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0911626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0912046Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0912468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0912890Z return func(*args, **kwargs) 2025-10-10T00:53:02.0913299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0913745Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0914189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0914607Z return func(*args, **kwargs) 2025-10-10T00:53:02.0915018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0915450Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0915603Z 2025-10-10T00:53:02.0915720Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0916126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0916494Z res = mod(**inputs) 2025-10-10T00:53:02.0916888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0917316Z outputs = self.model.decoder( 2025-10-10T00:53:02.0917733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0918164Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0918555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0918960Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0919372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0919782Z return func(*args, **kwargs) 2025-10-10T00:53:02.0920188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0920662Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0920852Z 2025-10-10T00:53:02.0920975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0921360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0921720Z res = mod(**inputs) 2025-10-10T00:53:02.0922193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0922609Z outputs = self.model.decoder( 2025-10-10T00:53:02.0923009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0923433Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0923821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0924227Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0924651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0925045Z return func(*args, **kwargs) 2025-10-10T00:53:02.0925449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0925917Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0926342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0926714Z return self.act(input) 2025-10-10T00:53:02.0926836Z 2025-10-10T00:53:02.0926950Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0927334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0927677Z res = mod(**inputs) 2025-10-10T00:53:02.0928063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0928472Z outputs = self.model.decoder( 2025-10-10T00:53:02.0928880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0929292Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0929676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0930070Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0930473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0930877Z return func(*args, **kwargs) 2025-10-10T00:53:02.0931282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.0931853Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.0932010Z 2025-10-10T00:53:02.0932133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0932531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0932879Z res = mod(**inputs) 2025-10-10T00:53:02.0933273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0933743Z outputs = self.model.decoder( 2025-10-10T00:53:02.0934464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0934936Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0935378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0935898Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0936390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0936875Z return func(*args, **kwargs) 2025-10-10T00:53:02.0937352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0937874Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0938507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0938998Z return func(*args, **kwargs) 2025-10-10T00:53:02.0939446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.0940025Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.0940307Z 2025-10-10T00:53:02.0940443Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0940904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0941348Z res = mod(**inputs) 2025-10-10T00:53:02.0941772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0942273Z outputs = self.model.decoder( 2025-10-10T00:53:02.0942817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0959506Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0960125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0960559Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0961004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0961447Z return func(*args, **kwargs) 2025-10-10T00:53:02.0961879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0962354Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0962780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0963201Z return func(*args, **kwargs) 2025-10-10T00:53:02.0963609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.0964049Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.0964211Z 2025-10-10T00:53:02.0964343Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0964751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0965129Z res = mod(**inputs) 2025-10-10T00:53:02.0965523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0965950Z outputs = self.model.decoder( 2025-10-10T00:53:02.0966367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0966778Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0967171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0967568Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0967983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0968379Z return func(*args, **kwargs) 2025-10-10T00:53:02.0968780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0969213Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0969617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0970000Z return func(*args, **kwargs) 2025-10-10T00:53:02.0970368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.0971002Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.0971161Z 2025-10-10T00:53:02.0971250Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.0971514Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0971900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0972256Z res = mod(**inputs) 2025-10-10T00:53:02.0972667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0973079Z outputs = self.model.decoder( 2025-10-10T00:53:02.0973506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0973927Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0974321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0974732Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0975151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0975566Z return func(*args, **kwargs) 2025-10-10T00:53:02.0975972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0976424Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0976864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0977278Z return func(*args, **kwargs) 2025-10-10T00:53:02.0977679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.0978135Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.0978643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.0979184Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.0979388Z 2025-10-10T00:53:02.0979515Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0979907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0980270Z res = mod(**inputs) 2025-10-10T00:53:02.0980668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0981099Z outputs = self.model.decoder( 2025-10-10T00:53:02.0981524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0981944Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0982343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0982773Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0983208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0983635Z return func(*args, **kwargs) 2025-10-10T00:53:02.0984058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.0984513Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.0984994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0985426Z return func(*args, **kwargs) 2025-10-10T00:53:02.0985842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.0986441Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.0986648Z 2025-10-10T00:53:02.0986768Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0987177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0987622Z res = mod(**inputs) 2025-10-10T00:53:02.0988009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0988430Z outputs = self.model.decoder( 2025-10-10T00:53:02.0988882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0989297Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0989671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0990068Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0990486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0990896Z return func(*args, **kwargs) 2025-10-10T00:53:02.0991299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0991760Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0991959Z 2025-10-10T00:53:02.0992076Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0992467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0992818Z res = mod(**inputs) 2025-10-10T00:53:02.0993211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0993638Z outputs = self.model.decoder( 2025-10-10T00:53:02.0994057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.0994483Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.0994865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.0995255Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.0995670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.0996082Z return func(*args, **kwargs) 2025-10-10T00:53:02.0996486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.0996959Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.0997380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.0997775Z return self.act(input) 2025-10-10T00:53:02.0997909Z 2025-10-10T00:53:02.0998030Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.0998430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.0998783Z res = mod(**inputs) 2025-10-10T00:53:02.0999167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.0999591Z outputs = self.model.decoder( 2025-10-10T00:53:02.1000002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1000442Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1000828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1001248Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1001715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1002155Z return func(*args, **kwargs) 2025-10-10T00:53:02.1002557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.1002974Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.1003133Z 2025-10-10T00:53:02.1003246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1003641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1003988Z res = mod(**inputs) 2025-10-10T00:53:02.1004367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1004782Z outputs = self.model.decoder( 2025-10-10T00:53:02.1005187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1005607Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1005987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1006374Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1006786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1007191Z return func(*args, **kwargs) 2025-10-10T00:53:02.1007590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1008030Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1008458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1008872Z return func(*args, **kwargs) 2025-10-10T00:53:02.1009286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.1009811Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.1010030Z 2025-10-10T00:53:02.1010154Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1010544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1010902Z res = mod(**inputs) 2025-10-10T00:53:02.1011303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1011736Z outputs = self.model.decoder( 2025-10-10T00:53:02.1012144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1012575Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1012972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1013386Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1013811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1014221Z return func(*args, **kwargs) 2025-10-10T00:53:02.1014632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1015098Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1015539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1015954Z return func(*args, **kwargs) 2025-10-10T00:53:02.1016368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.1016811Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.1017002Z 2025-10-10T00:53:02.1017176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1017581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1017935Z res = mod(**inputs) 2025-10-10T00:53:02.1018339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1018781Z outputs = self.model.decoder( 2025-10-10T00:53:02.1019204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1019644Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1020029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1020439Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1020864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1021284Z return func(*args, **kwargs) 2025-10-10T00:53:02.1021686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1022143Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1022584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1023010Z return func(*args, **kwargs) 2025-10-10T00:53:02.1023421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.1023870Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.1024039Z 2025-10-10T00:53:02.1024132Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.1024407Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1024812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1025175Z res = mod(**inputs) 2025-10-10T00:53:02.1025565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1026084Z outputs = self.model.decoder( 2025-10-10T00:53:02.1026547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1026989Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1027381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1027801Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1028241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1028668Z return func(*args, **kwargs) 2025-10-10T00:53:02.1029110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1029572Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1030028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1030457Z return func(*args, **kwargs) 2025-10-10T00:53:02.1030870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.1031323Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.1032005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.1032559Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.1032775Z 2025-10-10T00:53:02.1032996Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1033469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1033844Z res = mod(**inputs) 2025-10-10T00:53:02.1034249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1034670Z outputs = self.model.decoder( 2025-10-10T00:53:02.1035065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1035494Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1035888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1036287Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1036705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1037121Z return func(*args, **kwargs) 2025-10-10T00:53:02.1037528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1037977Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1038402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1038818Z return func(*args, **kwargs) 2025-10-10T00:53:02.1039216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.1039627Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.1039769Z 2025-10-10T00:53:02.1039886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1040253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1040594Z res = mod(**inputs) 2025-10-10T00:53:02.1040968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1041370Z outputs = self.model.decoder( 2025-10-10T00:53:02.1041755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1042156Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1042523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1042904Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1043303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1043707Z return func(*args, **kwargs) 2025-10-10T00:53:02.1044108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.1044589Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.1044785Z 2025-10-10T00:53:02.1044914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1045314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1045670Z res = mod(**inputs) 2025-10-10T00:53:02.1046064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1046486Z outputs = self.model.decoder( 2025-10-10T00:53:02.1046901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1047312Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1047675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1048114Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1048504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1048919Z return func(*args, **kwargs) 2025-10-10T00:53:02.1049323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.1049777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.1050194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.1050568Z return self.act(input) 2025-10-10T00:53:02.1050688Z 2025-10-10T00:53:02.1050801Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1051188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1051539Z res = mod(**inputs) 2025-10-10T00:53:02.1051934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1052354Z outputs = self.model.decoder( 2025-10-10T00:53:02.1052764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1053191Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1053580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1053991Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1054397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1054800Z return func(*args, **kwargs) 2025-10-10T00:53:02.1055200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.1055630Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.1055778Z 2025-10-10T00:53:02.1055900Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1056278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1056627Z res = mod(**inputs) 2025-10-10T00:53:02.1057011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1057450Z outputs = self.model.decoder( 2025-10-10T00:53:02.1057875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1058280Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1058658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1059051Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1059465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1059858Z return func(*args, **kwargs) 2025-10-10T00:53:02.1060256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1060697Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1061128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1061531Z return func(*args, **kwargs) 2025-10-10T00:53:02.1061924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-10-10T00:53:02.1062441Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:53:02.1062673Z 2025-10-10T00:53:02.1062790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1063292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1063672Z res = mod(**inputs) 2025-10-10T00:53:02.1064064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1064497Z outputs = self.model.decoder( 2025-10-10T00:53:02.1064927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1065369Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1065757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1066439Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1066874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1067302Z return func(*args, **kwargs) 2025-10-10T00:53:02.1067718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1068160Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1068585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1068967Z return func(*args, **kwargs) 2025-10-10T00:53:02.1069359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-10-10T00:53:02.1069789Z key_states = self.k_proj(current_states) 2025-10-10T00:53:02.1069941Z 2025-10-10T00:53:02.1070049Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1070419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1070752Z res = mod(**inputs) 2025-10-10T00:53:02.1071137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1071548Z outputs = self.model.decoder( 2025-10-10T00:53:02.1071950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1072343Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1072703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1073082Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1073468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1073850Z return func(*args, **kwargs) 2025-10-10T00:53:02.1074228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1074650Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1075059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1075431Z return func(*args, **kwargs) 2025-10-10T00:53:02.1075807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-10-10T00:53:02.1076215Z value_states = self.v_proj(current_states) 2025-10-10T00:53:02.1076360Z 2025-10-10T00:53:02.1076452Z cudagraph partition due to non gpu ops 2025-10-10T00:53:02.1076694Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1077064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1077392Z res = mod(**inputs) 2025-10-10T00:53:02.1077758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1078242Z outputs = self.model.decoder( 2025-10-10T00:53:02.1078623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1079017Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1079371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1079746Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1080126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1080507Z return func(*args, **kwargs) 2025-10-10T00:53:02.1080883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1081312Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1081752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1082151Z return func(*args, **kwargs) 2025-10-10T00:53:02.1082548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-10-10T00:53:02.1082665Z attn_output, attn_weights = attention_interface( 2025-10-10T00:53:02.1082981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:53:02.1083129Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:02.1083141Z 2025-10-10T00:53:02.1083256Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1083473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1083549Z res = mod(**inputs) 2025-10-10T00:53:02.1083821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1083913Z outputs = self.model.decoder( 2025-10-10T00:53:02.1084179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1084260Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1084507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1084594Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1084857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1084932Z return func(*args, **kwargs) 2025-10-10T00:53:02.1085196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-10-10T00:53:02.1085312Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:53:02.1085576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1085658Z return func(*args, **kwargs) 2025-10-10T00:53:02.1085923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-10-10T00:53:02.1086014Z attn_output = self.out_proj(attn_output) 2025-10-10T00:53:02.1086024Z 2025-10-10T00:53:02.1086136Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1086352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1086430Z res = mod(**inputs) 2025-10-10T00:53:02.1086698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1086784Z outputs = self.model.decoder( 2025-10-10T00:53:02.1087086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1087197Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1087443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1087530Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1087797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1087872Z return func(*args, **kwargs) 2025-10-10T00:53:02.1088142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.1088279Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.1088283Z 2025-10-10T00:53:02.1088394Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1088617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1088690Z res = mod(**inputs) 2025-10-10T00:53:02.1088964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1089045Z outputs = self.model.decoder( 2025-10-10T00:53:02.1089312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1089400Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1089639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1089733Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1089994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1090069Z return func(*args, **kwargs) 2025-10-10T00:53:02.1090352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-10-10T00:53:02.1090482Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:53:02.1090720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:02.1090798Z return self.act(input) 2025-10-10T00:53:02.1090801Z 2025-10-10T00:53:02.1090911Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1091129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1091198Z res = mod(**inputs) 2025-10-10T00:53:02.1091470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-10-10T00:53:02.1091570Z outputs = self.model.decoder( 2025-10-10T00:53:02.1091838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-10-10T00:53:02.1091930Z layer_outputs = decoder_layer( 2025-10-10T00:53:02.1092170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:02.1092256Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:02.1092526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:02.1092603Z return func(*args, **kwargs) 2025-10-10T00:53:02.1092877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-10-10T00:53:02.1092967Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:53:02.1092971Z 2025-10-10T00:53:02.1093088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1093301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1093406Z res = mod(**inputs) 2025-10-10T00:53:02.1093711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1918, in forward 2025-10-10T00:53:02.1093799Z logits = self.lm_head(outputs[0]) 2025-10-10T00:53:02.1093803Z 2025-10-10T00:53:02.1093918Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:02.1094130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:02.1094200Z res = mod(**inputs) 2025-10-10T00:53:02.1094475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1924, in forward 2025-10-10T00:53:02.1094635Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:53:02.1094640Z 2025-10-10T00:53:13.4084759Z Compilation time (from dynamo_timed): 17.119227959 2025-10-10T00:53:13.4294090Z pass 2025-10-10T00:53:13.4294704Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:13.4295702Z TIMING: _recursive_pre_grad_passes:0.00763 _recursive_joint_graph_passes:0.71099 _recursive_post_grad_passes:0.07982 async_compile.wait:0.84231 code_gen:9.76188 inductor_compile:11.1486 backend_compile:14.38085 gc:0.00134 entire_frame_compile:17.11923 total_wall_time:17.11923 2025-10-10T00:53:13.4296779Z STATS: call_* op count: 336 | FakeTensorMode.__torch_dispatch__:7450 | FakeTensor.__torch_dispatch__:4394 | ProxyTorchDispatchMode.__torch_dispatch__:2003 2025-10-10T00:53:13.4297347Z Dynamo produced 1 graphs covering 336 ops with 0 graph breaks (0 unique) 2025-10-10T00:53:16.1091628Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:53:16.1092508Z import pynvml # type: ignore[import] 2025-10-10T00:53:19.5345224Z 2025-10-10T00:53:20.7645816Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:53:20.7650350Z loading model: 0it [00:01, ?it/s] 2025-10-10T00:53:20.7661251Z cpu eval BertForMaskedLM 2025-10-10T00:53:21.3107070Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:21.5575629Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:21.8013487Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:29.8940782Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8941349Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8941723Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8942084Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8942456Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8942802Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8943224Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8943565Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8943924Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8944294Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8944643Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8945005Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8945440Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.8946214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.8946835Z res = mod(**inputs) 2025-10-10T00:53:29.8947527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.8948247Z outputs = self.bert( 2025-10-10T00:53:29.8948899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.8950188Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.8950916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.8951677Z layer_outputs = layer_module( 2025-10-10T00:53:29.8952343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.8952834Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.8953251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8953664Z return func(*args, **kwargs) 2025-10-10T00:53:29.8954069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.8954505Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.8955153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8955826Z return func(*args, **kwargs) 2025-10-10T00:53:29.8956490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.8957162Z self_outputs = self.self( 2025-10-10T00:53:29.8957818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8958486Z return func(*args, **kwargs) 2025-10-10T00:53:29.8959137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.8960101Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.8960589Z 2025-10-10T00:53:29.8960751Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.8961380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.8961912Z res = mod(**inputs) 2025-10-10T00:53:29.8962539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.8963178Z outputs = self.bert( 2025-10-10T00:53:29.8963767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.8964423Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.8965088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.8965743Z layer_outputs = layer_module( 2025-10-10T00:53:29.8966375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.8967020Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.8967663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8968344Z return func(*args, **kwargs) 2025-10-10T00:53:29.8968980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.8969665Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.8970337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8970971Z return func(*args, **kwargs) 2025-10-10T00:53:29.8971645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.8972295Z self_outputs = self.self( 2025-10-10T00:53:29.8972926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8973821Z return func(*args, **kwargs) 2025-10-10T00:53:29.8974484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.8975147Z self.key(current_states) 2025-10-10T00:53:29.8975358Z 2025-10-10T00:53:29.8975532Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.8976166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.8976733Z res = mod(**inputs) 2025-10-10T00:53:29.8977356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.8978068Z outputs = self.bert( 2025-10-10T00:53:29.8978703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.8979426Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.8980083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.8980759Z layer_outputs = layer_module( 2025-10-10T00:53:29.8981379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.8982015Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.8982673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8983340Z return func(*args, **kwargs) 2025-10-10T00:53:29.8983979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.8984701Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.8985398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8986232Z return func(*args, **kwargs) 2025-10-10T00:53:29.8986903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.8987586Z self_outputs = self.self( 2025-10-10T00:53:29.8988200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8988882Z return func(*args, **kwargs) 2025-10-10T00:53:29.8989513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.8990204Z self.value(current_states) 2025-10-10T00:53:29.8990406Z 2025-10-10T00:53:29.8990543Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.8990943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.8991562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.8992128Z res = mod(**inputs) 2025-10-10T00:53:29.8992822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.8993531Z outputs = self.bert( 2025-10-10T00:53:29.8994145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.8994782Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.8995433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.8996177Z layer_outputs = layer_module( 2025-10-10T00:53:29.8996873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.8997577Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.8998250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.8999113Z return func(*args, **kwargs) 2025-10-10T00:53:29.8999778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9000485Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9001191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9001873Z return func(*args, **kwargs) 2025-10-10T00:53:29.9002529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9003174Z self_outputs = self.self( 2025-10-10T00:53:29.9003823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9004509Z return func(*args, **kwargs) 2025-10-10T00:53:29.9005195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9006027Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9006360Z 2025-10-10T00:53:29.9006549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9007191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9007784Z res = mod(**inputs) 2025-10-10T00:53:29.9008433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9009160Z outputs = self.bert( 2025-10-10T00:53:29.9009812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9010513Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9011203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9011945Z layer_outputs = layer_module( 2025-10-10T00:53:29.9012622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9013306Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9014004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9014683Z return func(*args, **kwargs) 2025-10-10T00:53:29.9015362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9016100Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9016810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9017494Z return func(*args, **kwargs) 2025-10-10T00:53:29.9018191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9019063Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9019868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9020627Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9020901Z 2025-10-10T00:53:29.9021098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9021775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9022373Z res = mod(**inputs) 2025-10-10T00:53:29.9023040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9023770Z outputs = self.bert( 2025-10-10T00:53:29.9024624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9025456Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9026329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9027081Z layer_outputs = layer_module( 2025-10-10T00:53:29.9027775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9028450Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9029191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9029911Z return func(*args, **kwargs) 2025-10-10T00:53:29.9030627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9031396Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9032431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9033196Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9034001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9034883Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9035680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9036402Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9036651Z 2025-10-10T00:53:29.9036844Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9037498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9038088Z res = mod(**inputs) 2025-10-10T00:53:29.9038768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9039468Z outputs = self.bert( 2025-10-10T00:53:29.9040116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9040832Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9041536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9042235Z layer_outputs = layer_module( 2025-10-10T00:53:29.9042865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9043512Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9044203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9044875Z return func(*args, **kwargs) 2025-10-10T00:53:29.9045563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9046283Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9047033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9047757Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9048512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9049331Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9050093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9050842Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9051791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9052502Z return self.act(input) 2025-10-10T00:53:29.9052700Z 2025-10-10T00:53:29.9052893Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9053532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9054112Z res = mod(**inputs) 2025-10-10T00:53:29.9054769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9055473Z outputs = self.bert( 2025-10-10T00:53:29.9056124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9056817Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9057509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9058238Z layer_outputs = layer_module( 2025-10-10T00:53:29.9058881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9059539Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9060253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9060961Z return func(*args, **kwargs) 2025-10-10T00:53:29.9061687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9062458Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9063237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9064011Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9064821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9065756Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9066758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9067546Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9067819Z 2025-10-10T00:53:29.9068498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9069192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9069818Z res = mod(**inputs) 2025-10-10T00:53:29.9070518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9071260Z outputs = self.bert( 2025-10-10T00:53:29.9071963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9072729Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9073479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9074233Z layer_outputs = layer_module( 2025-10-10T00:53:29.9074920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9075630Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9076486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9077150Z return func(*args, **kwargs) 2025-10-10T00:53:29.9077861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9078643Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9079656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9080382Z return func(*args, **kwargs) 2025-10-10T00:53:29.9081092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9081817Z self_outputs = self.self( 2025-10-10T00:53:29.9082516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9083231Z return func(*args, **kwargs) 2025-10-10T00:53:29.9083905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9084877Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9085406Z 2025-10-10T00:53:29.9085599Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9086283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9086876Z res = mod(**inputs) 2025-10-10T00:53:29.9087555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9088254Z outputs = self.bert( 2025-10-10T00:53:29.9088942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9089692Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9090421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9091151Z layer_outputs = layer_module( 2025-10-10T00:53:29.9091828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9092522Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9093292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9094010Z return func(*args, **kwargs) 2025-10-10T00:53:29.9094720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9095485Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9096243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9096963Z return func(*args, **kwargs) 2025-10-10T00:53:29.9097662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9098406Z self_outputs = self.self( 2025-10-10T00:53:29.9099110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9099843Z return func(*args, **kwargs) 2025-10-10T00:53:29.9100559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9101285Z self.key(current_states) 2025-10-10T00:53:29.9101518Z 2025-10-10T00:53:29.9101720Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9102445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9103077Z res = mod(**inputs) 2025-10-10T00:53:29.9103785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9104544Z outputs = self.bert( 2025-10-10T00:53:29.9105251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9106116Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9107056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9107783Z layer_outputs = layer_module( 2025-10-10T00:53:29.9108456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9109157Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9109908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9110645Z return func(*args, **kwargs) 2025-10-10T00:53:29.9111363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9112118Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9112881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9113626Z return func(*args, **kwargs) 2025-10-10T00:53:29.9114299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9115008Z self_outputs = self.self( 2025-10-10T00:53:29.9115714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9116425Z return func(*args, **kwargs) 2025-10-10T00:53:29.9117119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9117812Z self.value(current_states) 2025-10-10T00:53:29.9118038Z 2025-10-10T00:53:29.9118187Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9118644Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9119351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9119976Z res = mod(**inputs) 2025-10-10T00:53:29.9120677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9121442Z outputs = self.bert( 2025-10-10T00:53:29.9122182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9122955Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9123708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9124454Z layer_outputs = layer_module( 2025-10-10T00:53:29.9125136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9125839Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9126589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9127290Z return func(*args, **kwargs) 2025-10-10T00:53:29.9128013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9128783Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9129536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9130266Z return func(*args, **kwargs) 2025-10-10T00:53:29.9130953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9131858Z self_outputs = self.self( 2025-10-10T00:53:29.9132567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9133298Z return func(*args, **kwargs) 2025-10-10T00:53:29.9134231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9135184Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9135552Z 2025-10-10T00:53:29.9135749Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9136445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9137059Z res = mod(**inputs) 2025-10-10T00:53:29.9137740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9138472Z outputs = self.bert( 2025-10-10T00:53:29.9139160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9139908Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9140637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9141385Z layer_outputs = layer_module( 2025-10-10T00:53:29.9142062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9142758Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9143505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9144235Z return func(*args, **kwargs) 2025-10-10T00:53:29.9144975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9145775Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9146640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9147397Z return func(*args, **kwargs) 2025-10-10T00:53:29.9148155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9148968Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9149766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9150493Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9150740Z 2025-10-10T00:53:29.9150920Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9151572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9152154Z res = mod(**inputs) 2025-10-10T00:53:29.9152807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9153496Z outputs = self.bert( 2025-10-10T00:53:29.9154139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9154852Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9155561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9156282Z layer_outputs = layer_module( 2025-10-10T00:53:29.9156952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9157624Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9158303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9158967Z return func(*args, **kwargs) 2025-10-10T00:53:29.9159626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9160327Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9161223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9161929Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9162677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9163510Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9164295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9165027Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9165280Z 2025-10-10T00:53:29.9165459Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9166105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9166676Z res = mod(**inputs) 2025-10-10T00:53:29.9167327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9168021Z outputs = self.bert( 2025-10-10T00:53:29.9168669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9169349Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9169943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9170640Z layer_outputs = layer_module( 2025-10-10T00:53:29.9171275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9171955Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9172669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9173352Z return func(*args, **kwargs) 2025-10-10T00:53:29.9174035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9174772Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9175502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9176230Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9176975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9177833Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9178629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9179411Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9180116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9180751Z return self.act(input) 2025-10-10T00:53:29.9180957Z 2025-10-10T00:53:29.9181138Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9181810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9182394Z res = mod(**inputs) 2025-10-10T00:53:29.9183040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9183728Z outputs = self.bert( 2025-10-10T00:53:29.9184383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9185082Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9185770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9186821Z layer_outputs = layer_module( 2025-10-10T00:53:29.9187497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9188159Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9188849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9189526Z return func(*args, **kwargs) 2025-10-10T00:53:29.9190212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9190937Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9191682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9192409Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9193167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9194028Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9194814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9195515Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9195756Z 2025-10-10T00:53:29.9195943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9196568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9197134Z res = mod(**inputs) 2025-10-10T00:53:29.9197765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9198443Z outputs = self.bert( 2025-10-10T00:53:29.9199079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9199771Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9200444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9201109Z layer_outputs = layer_module( 2025-10-10T00:53:29.9201742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9202390Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9203080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9203768Z return func(*args, **kwargs) 2025-10-10T00:53:29.9204447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9205130Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9205819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9206484Z return func(*args, **kwargs) 2025-10-10T00:53:29.9207135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9207822Z self_outputs = self.self( 2025-10-10T00:53:29.9208459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9209110Z return func(*args, **kwargs) 2025-10-10T00:53:29.9209760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9210693Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9211180Z 2025-10-10T00:53:29.9211469Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9212186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9212761Z res = mod(**inputs) 2025-10-10T00:53:29.9213423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9214105Z outputs = self.bert( 2025-10-10T00:53:29.9214757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9215440Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9216106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9216786Z layer_outputs = layer_module( 2025-10-10T00:53:29.9217414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9218063Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9218742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9219404Z return func(*args, **kwargs) 2025-10-10T00:53:29.9220076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9220783Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9221521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9222240Z return func(*args, **kwargs) 2025-10-10T00:53:29.9222965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9223707Z self_outputs = self.self( 2025-10-10T00:53:29.9224397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9225128Z return func(*args, **kwargs) 2025-10-10T00:53:29.9225835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9226678Z self.key(current_states) 2025-10-10T00:53:29.9226894Z 2025-10-10T00:53:29.9227099Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9227774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9228379Z res = mod(**inputs) 2025-10-10T00:53:29.9229073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9229808Z outputs = self.bert( 2025-10-10T00:53:29.9230488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9231213Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9232114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9232867Z layer_outputs = layer_module( 2025-10-10T00:53:29.9233543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9234250Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9234992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9235712Z return func(*args, **kwargs) 2025-10-10T00:53:29.9236440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9237205Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9238129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9238938Z return func(*args, **kwargs) 2025-10-10T00:53:29.9239636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9240345Z self_outputs = self.self( 2025-10-10T00:53:29.9240993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9241668Z return func(*args, **kwargs) 2025-10-10T00:53:29.9242379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9243116Z self.value(current_states) 2025-10-10T00:53:29.9243338Z 2025-10-10T00:53:29.9243504Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9243921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9244586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9245179Z res = mod(**inputs) 2025-10-10T00:53:29.9245836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9246557Z outputs = self.bert( 2025-10-10T00:53:29.9247248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9247973Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9248686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9249407Z layer_outputs = layer_module( 2025-10-10T00:53:29.9250043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9250739Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9251489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9252222Z return func(*args, **kwargs) 2025-10-10T00:53:29.9252885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9253591Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9254282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9254966Z return func(*args, **kwargs) 2025-10-10T00:53:29.9255648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9256336Z self_outputs = self.self( 2025-10-10T00:53:29.9256996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9257680Z return func(*args, **kwargs) 2025-10-10T00:53:29.9258369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9259187Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9259524Z 2025-10-10T00:53:29.9259708Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9260368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9260978Z res = mod(**inputs) 2025-10-10T00:53:29.9261661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9262418Z outputs = self.bert( 2025-10-10T00:53:29.9263119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9263864Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9264686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9265484Z layer_outputs = layer_module( 2025-10-10T00:53:29.9266258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9267002Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9267746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9268473Z return func(*args, **kwargs) 2025-10-10T00:53:29.9269194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9269955Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9270705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9271425Z return func(*args, **kwargs) 2025-10-10T00:53:29.9272143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9272977Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9273821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9274584Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9274827Z 2025-10-10T00:53:29.9275035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9275721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9276325Z res = mod(**inputs) 2025-10-10T00:53:29.9277009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9277752Z outputs = self.bert( 2025-10-10T00:53:29.9278463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9279215Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9279948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9280647Z layer_outputs = layer_module( 2025-10-10T00:53:29.9281314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9282018Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9282751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9283476Z return func(*args, **kwargs) 2025-10-10T00:53:29.9284150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9284863Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9285642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9286387Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9287139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9287983Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9288791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9289565Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9289812Z 2025-10-10T00:53:29.9289989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9290644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9291356Z res = mod(**inputs) 2025-10-10T00:53:29.9292096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9292832Z outputs = self.bert( 2025-10-10T00:53:29.9293504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9294203Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9294928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9295664Z layer_outputs = layer_module( 2025-10-10T00:53:29.9296325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9297029Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9297763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9298490Z return func(*args, **kwargs) 2025-10-10T00:53:29.9299192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9299950Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9300748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9301517Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9302316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9303197Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9304026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9304848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9305596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9306372Z return self.act(input) 2025-10-10T00:53:29.9306587Z 2025-10-10T00:53:29.9306781Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9307490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9308099Z res = mod(**inputs) 2025-10-10T00:53:29.9308787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9309517Z outputs = self.bert( 2025-10-10T00:53:29.9310215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9310988Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9311736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9312477Z layer_outputs = layer_module( 2025-10-10T00:53:29.9313154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9313877Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9314633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9315378Z return func(*args, **kwargs) 2025-10-10T00:53:29.9316115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9316896Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9317695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9318480Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9319465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9320413Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9321289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9322083Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9322367Z 2025-10-10T00:53:29.9322570Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9323293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9323920Z res = mod(**inputs) 2025-10-10T00:53:29.9324626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9325368Z outputs = self.bert( 2025-10-10T00:53:29.9326066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9326807Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9327534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9328256Z layer_outputs = layer_module( 2025-10-10T00:53:29.9328929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9329629Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9330353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9331043Z return func(*args, **kwargs) 2025-10-10T00:53:29.9331896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9332604Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9333285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9333960Z return func(*args, **kwargs) 2025-10-10T00:53:29.9334649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9335371Z self_outputs = self.self( 2025-10-10T00:53:29.9336074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9336778Z return func(*args, **kwargs) 2025-10-10T00:53:29.9337492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9338446Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9338892Z 2025-10-10T00:53:29.9339062Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9339681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9340223Z res = mod(**inputs) 2025-10-10T00:53:29.9340837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9341499Z outputs = self.bert( 2025-10-10T00:53:29.9342120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9342779Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9343419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9344095Z layer_outputs = layer_module( 2025-10-10T00:53:29.9344712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9345656Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9346449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9347161Z return func(*args, **kwargs) 2025-10-10T00:53:29.9347809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9348496Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9349166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9349815Z return func(*args, **kwargs) 2025-10-10T00:53:29.9350445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9351105Z self_outputs = self.self( 2025-10-10T00:53:29.9351745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9352392Z return func(*args, **kwargs) 2025-10-10T00:53:29.9353033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9353677Z self.key(current_states) 2025-10-10T00:53:29.9353869Z 2025-10-10T00:53:29.9354034Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9354629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9355155Z res = mod(**inputs) 2025-10-10T00:53:29.9355737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9356355Z outputs = self.bert( 2025-10-10T00:53:29.9356947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9357593Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9358205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9358838Z layer_outputs = layer_module( 2025-10-10T00:53:29.9359410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9360008Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9360640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9361243Z return func(*args, **kwargs) 2025-10-10T00:53:29.9361852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9362512Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9363162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9363771Z return func(*args, **kwargs) 2025-10-10T00:53:29.9364347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9364929Z self_outputs = self.self( 2025-10-10T00:53:29.9365467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9366029Z return func(*args, **kwargs) 2025-10-10T00:53:29.9366558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9367096Z self.value(current_states) 2025-10-10T00:53:29.9367264Z 2025-10-10T00:53:29.9367363Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9367674Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9368299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9368841Z res = mod(**inputs) 2025-10-10T00:53:29.9369392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9369991Z outputs = self.bert( 2025-10-10T00:53:29.9370519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9371132Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9371724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9372353Z layer_outputs = layer_module( 2025-10-10T00:53:29.9372935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9373537Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9374173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9374837Z return func(*args, **kwargs) 2025-10-10T00:53:29.9375478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9376132Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9376765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9377392Z return func(*args, **kwargs) 2025-10-10T00:53:29.9377947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9378511Z self_outputs = self.self( 2025-10-10T00:53:29.9379093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9379626Z return func(*args, **kwargs) 2025-10-10T00:53:29.9380176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9380884Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9381150Z 2025-10-10T00:53:29.9381304Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9381822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9382194Z res = mod(**inputs) 2025-10-10T00:53:29.9382560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9382945Z outputs = self.bert( 2025-10-10T00:53:29.9383301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9383691Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9384096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9384479Z layer_outputs = layer_module( 2025-10-10T00:53:29.9384841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9385223Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9385612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9386106Z return func(*args, **kwargs) 2025-10-10T00:53:29.9386591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9387234Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9387693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9388253Z return func(*args, **kwargs) 2025-10-10T00:53:29.9388618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9389048Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9389475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9389859Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9390008Z 2025-10-10T00:53:29.9390117Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9390475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9390801Z res = mod(**inputs) 2025-10-10T00:53:29.9391158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9391527Z outputs = self.bert( 2025-10-10T00:53:29.9391887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9392264Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9392634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9393011Z layer_outputs = layer_module( 2025-10-10T00:53:29.9393354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9393716Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9394090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9394459Z return func(*args, **kwargs) 2025-10-10T00:53:29.9394814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9395211Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9395616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9396006Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9396416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9396871Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9397301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9397697Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9397837Z 2025-10-10T00:53:29.9397955Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9398323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9398650Z res = mod(**inputs) 2025-10-10T00:53:29.9399015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9399401Z outputs = self.bert( 2025-10-10T00:53:29.9399766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9400145Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9400523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9400909Z layer_outputs = layer_module( 2025-10-10T00:53:29.9401264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9401631Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9402046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9402446Z return func(*args, **kwargs) 2025-10-10T00:53:29.9402809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9403201Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9403603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9403993Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9404401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9404856Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9405275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9405699Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9406097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9406435Z return self.act(input) 2025-10-10T00:53:29.9406549Z 2025-10-10T00:53:29.9406665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9407036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9407365Z res = mod(**inputs) 2025-10-10T00:53:29.9407720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9408098Z outputs = self.bert( 2025-10-10T00:53:29.9408449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9408830Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9409200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9409581Z layer_outputs = layer_module( 2025-10-10T00:53:29.9409934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9410299Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9410672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9411043Z return func(*args, **kwargs) 2025-10-10T00:53:29.9411410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9411806Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9412210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9412597Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9413011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9413472Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9413908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9414296Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9414434Z 2025-10-10T00:53:29.9414540Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9414898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9415218Z res = mod(**inputs) 2025-10-10T00:53:29.9415572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9415940Z outputs = self.bert( 2025-10-10T00:53:29.9416361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9416742Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9417117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9417500Z layer_outputs = layer_module( 2025-10-10T00:53:29.9417841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9418207Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9418587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9418959Z return func(*args, **kwargs) 2025-10-10T00:53:29.9419316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9419708Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9420087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9420458Z return func(*args, **kwargs) 2025-10-10T00:53:29.9420824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9421190Z self_outputs = self.self( 2025-10-10T00:53:29.9421552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9421920Z return func(*args, **kwargs) 2025-10-10T00:53:29.9422284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9422804Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9423090Z 2025-10-10T00:53:29.9423212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9423603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9423951Z res = mod(**inputs) 2025-10-10T00:53:29.9424340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9424749Z outputs = self.bert( 2025-10-10T00:53:29.9425128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9425537Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9425946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9426500Z layer_outputs = layer_module( 2025-10-10T00:53:29.9426895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9427333Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9427750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9428158Z return func(*args, **kwargs) 2025-10-10T00:53:29.9428566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9428986Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9429406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9429817Z return func(*args, **kwargs) 2025-10-10T00:53:29.9430217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9430621Z self_outputs = self.self( 2025-10-10T00:53:29.9431107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9431723Z return func(*args, **kwargs) 2025-10-10T00:53:29.9432162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9432578Z self.key(current_states) 2025-10-10T00:53:29.9432707Z 2025-10-10T00:53:29.9432826Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9433230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9433583Z res = mod(**inputs) 2025-10-10T00:53:29.9433975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9434388Z outputs = self.bert( 2025-10-10T00:53:29.9434770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9435197Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9435609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9436024Z layer_outputs = layer_module( 2025-10-10T00:53:29.9436397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9436798Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9437211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9437628Z return func(*args, **kwargs) 2025-10-10T00:53:29.9437996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9438379Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9438765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9439139Z return func(*args, **kwargs) 2025-10-10T00:53:29.9439504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9439874Z self_outputs = self.self( 2025-10-10T00:53:29.9440236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9440603Z return func(*args, **kwargs) 2025-10-10T00:53:29.9440968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9441351Z self.value(current_states) 2025-10-10T00:53:29.9441468Z 2025-10-10T00:53:29.9441553Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9441804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9442192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9442516Z res = mod(**inputs) 2025-10-10T00:53:29.9442871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9443251Z outputs = self.bert( 2025-10-10T00:53:29.9443609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9443994Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9444369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9444740Z layer_outputs = layer_module( 2025-10-10T00:53:29.9445094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9445461Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9445998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9446370Z return func(*args, **kwargs) 2025-10-10T00:53:29.9446729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9447122Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9447516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9447908Z return func(*args, **kwargs) 2025-10-10T00:53:29.9448289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9448671Z self_outputs = self.self( 2025-10-10T00:53:29.9449069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9449451Z return func(*args, **kwargs) 2025-10-10T00:53:29.9449824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9450267Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9450474Z 2025-10-10T00:53:29.9450588Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9450983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9451315Z res = mod(**inputs) 2025-10-10T00:53:29.9451679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9452058Z outputs = self.bert( 2025-10-10T00:53:29.9452424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9452812Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9453206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9453584Z layer_outputs = layer_module( 2025-10-10T00:53:29.9453940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9454316Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9454703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9455079Z return func(*args, **kwargs) 2025-10-10T00:53:29.9455455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9455843Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9456230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9456610Z return func(*args, **kwargs) 2025-10-10T00:53:29.9456976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9457421Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9457861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9458261Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9458403Z 2025-10-10T00:53:29.9458521Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9458879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9459209Z res = mod(**inputs) 2025-10-10T00:53:29.9459574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9459994Z outputs = self.bert( 2025-10-10T00:53:29.9460390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9460774Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9461155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9461542Z layer_outputs = layer_module( 2025-10-10T00:53:29.9461900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9462267Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9462656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9463051Z return func(*args, **kwargs) 2025-10-10T00:53:29.9463446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9463885Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9464320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9464750Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9465192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9465685Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9466287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9466729Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9466896Z 2025-10-10T00:53:29.9467016Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9467439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9467779Z res = mod(**inputs) 2025-10-10T00:53:29.9468144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9468537Z outputs = self.bert( 2025-10-10T00:53:29.9468906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9469304Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9469696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9470088Z layer_outputs = layer_module( 2025-10-10T00:53:29.9470440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9470809Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9471196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9471572Z return func(*args, **kwargs) 2025-10-10T00:53:29.9471932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9472330Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9472744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9473152Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9473566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9474032Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9474475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9474961Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9475344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9475677Z return self.act(input) 2025-10-10T00:53:29.9475799Z 2025-10-10T00:53:29.9475903Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9476263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9476586Z res = mod(**inputs) 2025-10-10T00:53:29.9476941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9477310Z outputs = self.bert( 2025-10-10T00:53:29.9477663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9478045Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9478427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9478800Z layer_outputs = layer_module( 2025-10-10T00:53:29.9479147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9479509Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9479887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9480257Z return func(*args, **kwargs) 2025-10-10T00:53:29.9480614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9481019Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9481431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9481845Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9482281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9482780Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9483226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9483625Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9483765Z 2025-10-10T00:53:29.9483892Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9484249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9484568Z res = mod(**inputs) 2025-10-10T00:53:29.9484924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9485306Z outputs = self.bert( 2025-10-10T00:53:29.9485663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9486047Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9486430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9486820Z layer_outputs = layer_module( 2025-10-10T00:53:29.9487178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9487553Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9487936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9488320Z return func(*args, **kwargs) 2025-10-10T00:53:29.9488741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9489173Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9489562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9489945Z return func(*args, **kwargs) 2025-10-10T00:53:29.9490318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9490773Z self_outputs = self.self( 2025-10-10T00:53:29.9491142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9491512Z return func(*args, **kwargs) 2025-10-10T00:53:29.9491886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9492417Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9492691Z 2025-10-10T00:53:29.9492806Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9493172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9493495Z res = mod(**inputs) 2025-10-10T00:53:29.9493861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9494247Z outputs = self.bert( 2025-10-10T00:53:29.9494607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9494992Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9495375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9495762Z layer_outputs = layer_module( 2025-10-10T00:53:29.9496121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9496499Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9496882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9497263Z return func(*args, **kwargs) 2025-10-10T00:53:29.9497639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9498042Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9498432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9498806Z return func(*args, **kwargs) 2025-10-10T00:53:29.9499181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9499568Z self_outputs = self.self( 2025-10-10T00:53:29.9499943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9500312Z return func(*args, **kwargs) 2025-10-10T00:53:29.9500685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9501075Z self.key(current_states) 2025-10-10T00:53:29.9501194Z 2025-10-10T00:53:29.9501309Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9501677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9502008Z res = mod(**inputs) 2025-10-10T00:53:29.9502399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9502806Z outputs = self.bert( 2025-10-10T00:53:29.9503230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9503666Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9504077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9504491Z layer_outputs = layer_module( 2025-10-10T00:53:29.9504874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9505272Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9505678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9506199Z return func(*args, **kwargs) 2025-10-10T00:53:29.9506614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9507053Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9507494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9507909Z return func(*args, **kwargs) 2025-10-10T00:53:29.9508321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9508746Z self_outputs = self.self( 2025-10-10T00:53:29.9509141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9509538Z return func(*args, **kwargs) 2025-10-10T00:53:29.9510052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9510471Z self.value(current_states) 2025-10-10T00:53:29.9510602Z 2025-10-10T00:53:29.9510704Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9510976Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9511379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9511748Z res = mod(**inputs) 2025-10-10T00:53:29.9512133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9512537Z outputs = self.bert( 2025-10-10T00:53:29.9512916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9513328Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9513739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9514164Z layer_outputs = layer_module( 2025-10-10T00:53:29.9514558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9514965Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9515402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9515829Z return func(*args, **kwargs) 2025-10-10T00:53:29.9516242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9516672Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9517087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9517493Z return func(*args, **kwargs) 2025-10-10T00:53:29.9517903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9518321Z self_outputs = self.self( 2025-10-10T00:53:29.9518713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9519203Z return func(*args, **kwargs) 2025-10-10T00:53:29.9519631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9520122Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9520330Z 2025-10-10T00:53:29.9520457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9520853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9521309Z res = mod(**inputs) 2025-10-10T00:53:29.9521760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9522194Z outputs = self.bert( 2025-10-10T00:53:29.9522585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9523015Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9523436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9523862Z layer_outputs = layer_module( 2025-10-10T00:53:29.9524255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9524656Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9525081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9525502Z return func(*args, **kwargs) 2025-10-10T00:53:29.9525914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9526352Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9526772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9527187Z return func(*args, **kwargs) 2025-10-10T00:53:29.9527596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9528077Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9528552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9529019Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9529241Z 2025-10-10T00:53:29.9529414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9530022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9530565Z res = mod(**inputs) 2025-10-10T00:53:29.9531175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9531302Z outputs = self.bert( 2025-10-10T00:53:29.9531922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9532051Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9532483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9532595Z layer_outputs = layer_module( 2025-10-10T00:53:29.9532987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9533115Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9533542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9533654Z return func(*args, **kwargs) 2025-10-10T00:53:29.9534267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9534494Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9534986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9535108Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9535565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9535767Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9536220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9536348Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9536354Z 2025-10-10T00:53:29.9536529Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9536857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9536959Z res = mod(**inputs) 2025-10-10T00:53:29.9537323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9537409Z outputs = self.bert( 2025-10-10T00:53:29.9537657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9537735Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9537991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9538065Z layer_outputs = layer_module( 2025-10-10T00:53:29.9538297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9538379Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9538628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9538711Z return func(*args, **kwargs) 2025-10-10T00:53:29.9538957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9539052Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9539318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9539403Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9539705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9539835Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9540105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9540235Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9540475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9540553Z return self.act(input) 2025-10-10T00:53:29.9540557Z 2025-10-10T00:53:29.9540671Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9540894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9540964Z res = mod(**inputs) 2025-10-10T00:53:29.9541240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9541315Z outputs = self.bert( 2025-10-10T00:53:29.9541586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9541677Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9542045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9542135Z layer_outputs = layer_module( 2025-10-10T00:53:29.9542381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9542470Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9542744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9542823Z return func(*args, **kwargs) 2025-10-10T00:53:29.9543098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9543192Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9543486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9543579Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9543887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9544047Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9544320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9544421Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9544426Z 2025-10-10T00:53:29.9544545Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9544768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9544850Z res = mod(**inputs) 2025-10-10T00:53:29.9545126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9545215Z outputs = self.bert( 2025-10-10T00:53:29.9545487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9545576Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9545847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9545927Z layer_outputs = layer_module( 2025-10-10T00:53:29.9546276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9546372Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9546648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9546727Z return func(*args, **kwargs) 2025-10-10T00:53:29.9547004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9547111Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9547377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9547465Z return func(*args, **kwargs) 2025-10-10T00:53:29.9547739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9547818Z self_outputs = self.self( 2025-10-10T00:53:29.9548084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9548156Z return func(*args, **kwargs) 2025-10-10T00:53:29.9548412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9548663Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9549819Z 2025-10-10T00:53:29.9549947Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9550151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9550218Z res = mod(**inputs) 2025-10-10T00:53:29.9550492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9550565Z outputs = self.bert( 2025-10-10T00:53:29.9550838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9550916Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9551177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9551263Z layer_outputs = layer_module( 2025-10-10T00:53:29.9551506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9551600Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9551863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9551947Z return func(*args, **kwargs) 2025-10-10T00:53:29.9552255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9552341Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9552591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9552661Z return func(*args, **kwargs) 2025-10-10T00:53:29.9552916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9552994Z self_outputs = self.self( 2025-10-10T00:53:29.9553238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9553315Z return func(*args, **kwargs) 2025-10-10T00:53:29.9553565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9553645Z self.key(current_states) 2025-10-10T00:53:29.9553648Z 2025-10-10T00:53:29.9553753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9553955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9554029Z res = mod(**inputs) 2025-10-10T00:53:29.9554283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9554361Z outputs = self.bert( 2025-10-10T00:53:29.9554613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9554699Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9554950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9555024Z layer_outputs = layer_module( 2025-10-10T00:53:29.9555256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9555338Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9555605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9555679Z return func(*args, **kwargs) 2025-10-10T00:53:29.9555941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9556039Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9556370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9556455Z return func(*args, **kwargs) 2025-10-10T00:53:29.9556716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9556793Z self_outputs = self.self( 2025-10-10T00:53:29.9557053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9557128Z return func(*args, **kwargs) 2025-10-10T00:53:29.9557397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9557474Z self.value(current_states) 2025-10-10T00:53:29.9557478Z 2025-10-10T00:53:29.9557576Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9557689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9557913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9557988Z res = mod(**inputs) 2025-10-10T00:53:29.9558239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9558313Z outputs = self.bert( 2025-10-10T00:53:29.9558559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9558634Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9558889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9558963Z layer_outputs = layer_module( 2025-10-10T00:53:29.9559207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9559300Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9559555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9559640Z return func(*args, **kwargs) 2025-10-10T00:53:29.9559902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9559997Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9560253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9560334Z return func(*args, **kwargs) 2025-10-10T00:53:29.9560596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9560672Z self_outputs = self.self( 2025-10-10T00:53:29.9560939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9561017Z return func(*args, **kwargs) 2025-10-10T00:53:29.9561287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9561434Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9561438Z 2025-10-10T00:53:29.9561550Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9561770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9561839Z res = mod(**inputs) 2025-10-10T00:53:29.9562115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9562187Z outputs = self.bert( 2025-10-10T00:53:29.9562450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9562616Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9562879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9562964Z layer_outputs = layer_module( 2025-10-10T00:53:29.9563201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9563293Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9563551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9563623Z return func(*args, **kwargs) 2025-10-10T00:53:29.9563895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9563983Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9564250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9564325Z return func(*args, **kwargs) 2025-10-10T00:53:29.9564587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9564736Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9565001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9565099Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9565103Z 2025-10-10T00:53:29.9565218Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9565441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9565512Z res = mod(**inputs) 2025-10-10T00:53:29.9565789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9565874Z outputs = self.bert( 2025-10-10T00:53:29.9566143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9566232Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9566501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9566580Z layer_outputs = layer_module( 2025-10-10T00:53:29.9566839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9566923Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9567186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9567260Z return func(*args, **kwargs) 2025-10-10T00:53:29.9567523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9567626Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9567904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9567997Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9568296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9568437Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9568700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9568792Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9568795Z 2025-10-10T00:53:29.9568915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9569209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9569290Z res = mod(**inputs) 2025-10-10T00:53:29.9569566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9569641Z outputs = self.bert( 2025-10-10T00:53:29.9569926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9570009Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9570287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9570366Z layer_outputs = layer_module( 2025-10-10T00:53:29.9570618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9570706Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9570979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9571076Z return func(*args, **kwargs) 2025-10-10T00:53:29.9571339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9571435Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9571713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9571797Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9572100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9572233Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9572507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9572634Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9572873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9572951Z return self.act(input) 2025-10-10T00:53:29.9572955Z 2025-10-10T00:53:29.9573069Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9573292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9573364Z res = mod(**inputs) 2025-10-10T00:53:29.9573638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9573711Z outputs = self.bert( 2025-10-10T00:53:29.9573973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9574067Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9574331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9574419Z layer_outputs = layer_module( 2025-10-10T00:53:29.9574661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9574751Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9575034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9575113Z return func(*args, **kwargs) 2025-10-10T00:53:29.9575391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9575485Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9575809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9575931Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9576245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9576402Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9576675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9576775Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9576778Z 2025-10-10T00:53:29.9576893Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9577122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9577199Z res = mod(**inputs) 2025-10-10T00:53:29.9577470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9577555Z outputs = self.bert( 2025-10-10T00:53:29.9577825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9577907Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9578186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9578266Z layer_outputs = layer_module( 2025-10-10T00:53:29.9578515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9578604Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9578875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9578952Z return func(*args, **kwargs) 2025-10-10T00:53:29.9579225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9579329Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9579593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9579678Z return func(*args, **kwargs) 2025-10-10T00:53:29.9579949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9580027Z self_outputs = self.self( 2025-10-10T00:53:29.9580302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9580378Z return func(*args, **kwargs) 2025-10-10T00:53:29.9580656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9580890Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9580898Z 2025-10-10T00:53:29.9581022Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9581239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9581309Z res = mod(**inputs) 2025-10-10T00:53:29.9581593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9581666Z outputs = self.bert( 2025-10-10T00:53:29.9581943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9582025Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9582294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9582382Z layer_outputs = layer_module( 2025-10-10T00:53:29.9582689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9582784Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9583048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9583123Z return func(*args, **kwargs) 2025-10-10T00:53:29.9583400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9583491Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9583762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9583838Z return func(*args, **kwargs) 2025-10-10T00:53:29.9584117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9584202Z self_outputs = self.self( 2025-10-10T00:53:29.9584467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9584552Z return func(*args, **kwargs) 2025-10-10T00:53:29.9584824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9584911Z self.key(current_states) 2025-10-10T00:53:29.9584915Z 2025-10-10T00:53:29.9585031Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9585246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9585326Z res = mod(**inputs) 2025-10-10T00:53:29.9585599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9585682Z outputs = self.bert( 2025-10-10T00:53:29.9586062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9586184Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9586465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9586546Z layer_outputs = layer_module( 2025-10-10T00:53:29.9586801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9586891Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9587165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9587242Z return func(*args, **kwargs) 2025-10-10T00:53:29.9587512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9587621Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9587889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9587975Z return func(*args, **kwargs) 2025-10-10T00:53:29.9588250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9588328Z self_outputs = self.self( 2025-10-10T00:53:29.9588604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9588682Z return func(*args, **kwargs) 2025-10-10T00:53:29.9588964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9589045Z self.value(current_states) 2025-10-10T00:53:29.9589049Z 2025-10-10T00:53:29.9589169Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9589369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9589591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9589673Z res = mod(**inputs) 2025-10-10T00:53:29.9589948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9590030Z outputs = self.bert( 2025-10-10T00:53:29.9590302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9590385Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9590666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9590748Z layer_outputs = layer_module( 2025-10-10T00:53:29.9591004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9591101Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9591384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9591469Z return func(*args, **kwargs) 2025-10-10T00:53:29.9591740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9591839Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9592101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9592180Z return func(*args, **kwargs) 2025-10-10T00:53:29.9592462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9592541Z self_outputs = self.self( 2025-10-10T00:53:29.9592818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9592898Z return func(*args, **kwargs) 2025-10-10T00:53:29.9593177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9593335Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9593338Z 2025-10-10T00:53:29.9593444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9593653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9593719Z res = mod(**inputs) 2025-10-10T00:53:29.9593992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9594065Z outputs = self.bert( 2025-10-10T00:53:29.9594334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9594424Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9594689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9594773Z layer_outputs = layer_module( 2025-10-10T00:53:29.9595010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9595103Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9595361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9595433Z return func(*args, **kwargs) 2025-10-10T00:53:29.9595690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9595775Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9596060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9596158Z return func(*args, **kwargs) 2025-10-10T00:53:29.9596411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9596555Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9596811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9596908Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9596912Z 2025-10-10T00:53:29.9597025Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9597246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9597316Z res = mod(**inputs) 2025-10-10T00:53:29.9597607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9597690Z outputs = self.bert( 2025-10-10T00:53:29.9597955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9598043Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9598308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9598385Z layer_outputs = layer_module( 2025-10-10T00:53:29.9598631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9598716Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9598984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9599058Z return func(*args, **kwargs) 2025-10-10T00:53:29.9599336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9599435Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9599713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9599806Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9600103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9600242Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9600506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9600596Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9600600Z 2025-10-10T00:53:29.9600718Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9600935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9601014Z res = mod(**inputs) 2025-10-10T00:53:29.9601283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9601356Z outputs = self.bert( 2025-10-10T00:53:29.9601630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9601710Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9601983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9602059Z layer_outputs = layer_module( 2025-10-10T00:53:29.9602301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9602394Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9602716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9602801Z return func(*args, **kwargs) 2025-10-10T00:53:29.9603063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9603162Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9603441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9603524Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9603830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9603959Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9604236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9604363Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9604594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9604679Z return self.act(input) 2025-10-10T00:53:29.9604683Z 2025-10-10T00:53:29.9604794Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9605021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9605092Z res = mod(**inputs) 2025-10-10T00:53:29.9605367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9605440Z outputs = self.bert( 2025-10-10T00:53:29.9605703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9605798Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9606062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9606149Z layer_outputs = layer_module( 2025-10-10T00:53:29.9606389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9606475Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9606740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9606816Z return func(*args, **kwargs) 2025-10-10T00:53:29.9607090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9607180Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9607463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9607556Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9607859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9608018Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9608294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9608393Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9608397Z 2025-10-10T00:53:29.9608511Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9608740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9608820Z res = mod(**inputs) 2025-10-10T00:53:29.9609119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9609231Z outputs = self.bert( 2025-10-10T00:53:29.9609507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9609590Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9609879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9609969Z layer_outputs = layer_module( 2025-10-10T00:53:29.9610220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9610307Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9610583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9610660Z return func(*args, **kwargs) 2025-10-10T00:53:29.9610946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9611050Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9611333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9611416Z return func(*args, **kwargs) 2025-10-10T00:53:29.9611695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9611773Z self_outputs = self.self( 2025-10-10T00:53:29.9612065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9612141Z return func(*args, **kwargs) 2025-10-10T00:53:29.9612426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9612665Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9612669Z 2025-10-10T00:53:29.9612793Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9613017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9613087Z res = mod(**inputs) 2025-10-10T00:53:29.9613379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9613454Z outputs = self.bert( 2025-10-10T00:53:29.9613740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9613822Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9614102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9614195Z layer_outputs = layer_module( 2025-10-10T00:53:29.9614447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9614541Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9614822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9614898Z return func(*args, **kwargs) 2025-10-10T00:53:29.9615183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9615274Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9615555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9615631Z return func(*args, **kwargs) 2025-10-10T00:53:29.9615960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9616069Z self_outputs = self.self( 2025-10-10T00:53:29.9616347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9616433Z return func(*args, **kwargs) 2025-10-10T00:53:29.9616704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9616792Z self.key(current_states) 2025-10-10T00:53:29.9616796Z 2025-10-10T00:53:29.9616910Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9617126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9617207Z res = mod(**inputs) 2025-10-10T00:53:29.9617482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9617566Z outputs = self.bert( 2025-10-10T00:53:29.9617846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9617930Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9618209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9618289Z layer_outputs = layer_module( 2025-10-10T00:53:29.9618540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9618629Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9618910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9618989Z return func(*args, **kwargs) 2025-10-10T00:53:29.9619260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9619365Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9619647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9619730Z return func(*args, **kwargs) 2025-10-10T00:53:29.9620001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9620079Z self_outputs = self.self( 2025-10-10T00:53:29.9620363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9620440Z return func(*args, **kwargs) 2025-10-10T00:53:29.9620721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9620801Z self.value(current_states) 2025-10-10T00:53:29.9620805Z 2025-10-10T00:53:29.9620898Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9621029Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9621248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9621328Z res = mod(**inputs) 2025-10-10T00:53:29.9621603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9621686Z outputs = self.bert( 2025-10-10T00:53:29.9621959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9622042Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9622321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9622401Z layer_outputs = layer_module( 2025-10-10T00:53:29.9622685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9622804Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9623070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9623156Z return func(*args, **kwargs) 2025-10-10T00:53:29.9623426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9623524Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9623788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9623865Z return func(*args, **kwargs) 2025-10-10T00:53:29.9624145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9624223Z self_outputs = self.self( 2025-10-10T00:53:29.9624502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9624578Z return func(*args, **kwargs) 2025-10-10T00:53:29.9624856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9625005Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9625009Z 2025-10-10T00:53:29.9625124Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9625351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9625424Z res = mod(**inputs) 2025-10-10T00:53:29.9625706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9625780Z outputs = self.bert( 2025-10-10T00:53:29.9626174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9626276Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9626550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9626638Z layer_outputs = layer_module( 2025-10-10T00:53:29.9626885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9626973Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9627247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9627325Z return func(*args, **kwargs) 2025-10-10T00:53:29.9627603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9627696Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9627976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9628054Z return func(*args, **kwargs) 2025-10-10T00:53:29.9628327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9628484Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9628758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9628871Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9628875Z 2025-10-10T00:53:29.9628986Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9629198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9629277Z res = mod(**inputs) 2025-10-10T00:53:29.9629587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9629695Z outputs = self.bert( 2025-10-10T00:53:29.9629958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9630044Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9630306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9630384Z layer_outputs = layer_module( 2025-10-10T00:53:29.9630631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9630715Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9630981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9631054Z return func(*args, **kwargs) 2025-10-10T00:53:29.9631324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9631423Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9631977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9632075Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9632372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9632505Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9632773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9632863Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9632869Z 2025-10-10T00:53:29.9632993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9633208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9633288Z res = mod(**inputs) 2025-10-10T00:53:29.9633554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9633629Z outputs = self.bert( 2025-10-10T00:53:29.9633901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9633980Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9634249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9634327Z layer_outputs = layer_module( 2025-10-10T00:53:29.9634566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9634669Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9634927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9635011Z return func(*args, **kwargs) 2025-10-10T00:53:29.9635277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9635375Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9635653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9635738Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9636042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9636172Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9636585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9636712Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9636950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9637038Z return self.act(input) 2025-10-10T00:53:29.9637042Z 2025-10-10T00:53:29.9637155Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9637380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9637452Z res = mod(**inputs) 2025-10-10T00:53:29.9637724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9637812Z outputs = self.bert( 2025-10-10T00:53:29.9638090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9638185Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9638461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9638552Z layer_outputs = layer_module( 2025-10-10T00:53:29.9638799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9638888Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9639174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9639250Z return func(*args, **kwargs) 2025-10-10T00:53:29.9639523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9639615Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9639897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9639992Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9640291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9640445Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9640712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9640809Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9640813Z 2025-10-10T00:53:29.9640925Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9641138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9641216Z res = mod(**inputs) 2025-10-10T00:53:29.9641488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9641570Z outputs = self.bert( 2025-10-10T00:53:29.9641841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9641919Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9642191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9642268Z layer_outputs = layer_module( 2025-10-10T00:53:29.9642512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9642599Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9642859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9642942Z return func(*args, **kwargs) 2025-10-10T00:53:29.9643280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9643380Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9643640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9643724Z return func(*args, **kwargs) 2025-10-10T00:53:29.9643987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9644063Z self_outputs = self.self( 2025-10-10T00:53:29.9644330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9644406Z return func(*args, **kwargs) 2025-10-10T00:53:29.9644678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9644906Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9644910Z 2025-10-10T00:53:29.9645023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9645245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9645314Z res = mod(**inputs) 2025-10-10T00:53:29.9645591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9645664Z outputs = self.bert( 2025-10-10T00:53:29.9645936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9646014Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9646276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9646368Z layer_outputs = layer_module( 2025-10-10T00:53:29.9646604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9646695Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9646958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9647034Z return func(*args, **kwargs) 2025-10-10T00:53:29.9647311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9647402Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9647684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9647759Z return func(*args, **kwargs) 2025-10-10T00:53:29.9648025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9648112Z self_outputs = self.self( 2025-10-10T00:53:29.9648372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9648455Z return func(*args, **kwargs) 2025-10-10T00:53:29.9648718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9648803Z self.key(current_states) 2025-10-10T00:53:29.9648807Z 2025-10-10T00:53:29.9648919Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9649133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9649210Z res = mod(**inputs) 2025-10-10T00:53:29.9649479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9649621Z outputs = self.bert( 2025-10-10T00:53:29.9649887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9649966Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9650237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9650313Z layer_outputs = layer_module( 2025-10-10T00:53:29.9650558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9650642Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9650901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9650983Z return func(*args, **kwargs) 2025-10-10T00:53:29.9651249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9651351Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9651611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9651694Z return func(*args, **kwargs) 2025-10-10T00:53:29.9651966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9652044Z self_outputs = self.self( 2025-10-10T00:53:29.9652315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9652390Z return func(*args, **kwargs) 2025-10-10T00:53:29.9652669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9652750Z self.value(current_states) 2025-10-10T00:53:29.9652754Z 2025-10-10T00:53:29.9652854Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9652980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9653210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9653301Z res = mod(**inputs) 2025-10-10T00:53:29.9653575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9653649Z outputs = self.bert( 2025-10-10T00:53:29.9653927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9654006Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9654296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9654372Z layer_outputs = layer_module( 2025-10-10T00:53:29.9654617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9654704Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9654964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9655048Z return func(*args, **kwargs) 2025-10-10T00:53:29.9655366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9655466Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9655738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9655814Z return func(*args, **kwargs) 2025-10-10T00:53:29.9656093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9656170Z self_outputs = self.self( 2025-10-10T00:53:29.9656514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9656592Z return func(*args, **kwargs) 2025-10-10T00:53:29.9656864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9657022Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9657026Z 2025-10-10T00:53:29.9657146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9657385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9657456Z res = mod(**inputs) 2025-10-10T00:53:29.9657742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9657817Z outputs = self.bert( 2025-10-10T00:53:29.9658092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9658186Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9658457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9658543Z layer_outputs = layer_module( 2025-10-10T00:53:29.9658786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9658876Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9659159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9659235Z return func(*args, **kwargs) 2025-10-10T00:53:29.9659515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9659608Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9659886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9659961Z return func(*args, **kwargs) 2025-10-10T00:53:29.9660234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9660387Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9660658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9660759Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9660763Z 2025-10-10T00:53:29.9660878Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9661111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9661191Z res = mod(**inputs) 2025-10-10T00:53:29.9661473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9661552Z outputs = self.bert( 2025-10-10T00:53:29.9661823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9661904Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9662182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9662261Z layer_outputs = layer_module( 2025-10-10T00:53:29.9662511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9662598Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9662872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9663026Z return func(*args, **kwargs) 2025-10-10T00:53:29.9663297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9663400Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9663688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9663780Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9664084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9664218Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9664494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9664587Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9664591Z 2025-10-10T00:53:29.9664718Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9664934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9665016Z res = mod(**inputs) 2025-10-10T00:53:29.9665289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9665364Z outputs = self.bert( 2025-10-10T00:53:29.9665639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9665719Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9666088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9666180Z layer_outputs = layer_module( 2025-10-10T00:53:29.9666426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9666533Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9666799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9666886Z return func(*args, **kwargs) 2025-10-10T00:53:29.9667156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9667252Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9667548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9667635Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9667947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9668083Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9668367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9668495Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9668732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9668821Z return self.act(input) 2025-10-10T00:53:29.9668825Z 2025-10-10T00:53:29.9668941Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9669169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9669242Z res = mod(**inputs) 2025-10-10T00:53:29.9669519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9669603Z outputs = self.bert( 2025-10-10T00:53:29.9669917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9670039Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9670313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9670401Z layer_outputs = layer_module( 2025-10-10T00:53:29.9670648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9670737Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9671010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9671087Z return func(*args, **kwargs) 2025-10-10T00:53:29.9671373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9671464Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9671750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9671840Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9672137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9672288Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9672554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9672646Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9672657Z 2025-10-10T00:53:29.9672768Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9672979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9673059Z res = mod(**inputs) 2025-10-10T00:53:29.9673335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9673418Z outputs = self.bert( 2025-10-10T00:53:29.9673684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9673762Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9674039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9674112Z layer_outputs = layer_module( 2025-10-10T00:53:29.9674344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9674425Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9674676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9674760Z return func(*args, **kwargs) 2025-10-10T00:53:29.9675007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9675097Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9675334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9675410Z return func(*args, **kwargs) 2025-10-10T00:53:29.9675654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9675725Z self_outputs = self.self( 2025-10-10T00:53:29.9675977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9676048Z return func(*args, **kwargs) 2025-10-10T00:53:29.9676346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9676582Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9676585Z 2025-10-10T00:53:29.9676691Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9676899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9676966Z res = mod(**inputs) 2025-10-10T00:53:29.9677225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9677293Z outputs = self.bert( 2025-10-10T00:53:29.9677539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9677621Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9677874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9677957Z layer_outputs = layer_module( 2025-10-10T00:53:29.9678180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9678268Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9678510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9678579Z return func(*args, **kwargs) 2025-10-10T00:53:29.9678836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9678919Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9679168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9679237Z return func(*args, **kwargs) 2025-10-10T00:53:29.9679500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9679585Z self_outputs = self.self( 2025-10-10T00:53:29.9679841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9679922Z return func(*args, **kwargs) 2025-10-10T00:53:29.9680187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9680263Z self.key(current_states) 2025-10-10T00:53:29.9680274Z 2025-10-10T00:53:29.9680384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9680595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9680676Z res = mod(**inputs) 2025-10-10T00:53:29.9680940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9681028Z outputs = self.bert( 2025-10-10T00:53:29.9681294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9681374Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9681645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9681723Z layer_outputs = layer_module( 2025-10-10T00:53:29.9681973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9682055Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9682296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9682374Z return func(*args, **kwargs) 2025-10-10T00:53:29.9682655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9682777Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9683019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9683100Z return func(*args, **kwargs) 2025-10-10T00:53:29.9683364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9683438Z self_outputs = self.self( 2025-10-10T00:53:29.9683712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9683783Z return func(*args, **kwargs) 2025-10-10T00:53:29.9684042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9684115Z self.value(current_states) 2025-10-10T00:53:29.9684122Z 2025-10-10T00:53:29.9684210Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9684326Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9684535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9684614Z res = mod(**inputs) 2025-10-10T00:53:29.9684882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9684955Z outputs = self.bert( 2025-10-10T00:53:29.9685269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9685370Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9685752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9685863Z layer_outputs = layer_module( 2025-10-10T00:53:29.9686220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9686341Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9686747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9686863Z return func(*args, **kwargs) 2025-10-10T00:53:29.9687263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9687392Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9687788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9687889Z return func(*args, **kwargs) 2025-10-10T00:53:29.9688296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9688406Z self_outputs = self.self( 2025-10-10T00:53:29.9688837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9688952Z return func(*args, **kwargs) 2025-10-10T00:53:29.9689400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9689621Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9689626Z 2025-10-10T00:53:29.9689797Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9690156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9690252Z res = mod(**inputs) 2025-10-10T00:53:29.9690660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9690757Z outputs = self.bert( 2025-10-10T00:53:29.9691327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9691445Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9691887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9692010Z layer_outputs = layer_module( 2025-10-10T00:53:29.9692405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9692517Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9692940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9693044Z return func(*args, **kwargs) 2025-10-10T00:53:29.9693498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9693643Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9694078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9694186Z return func(*args, **kwargs) 2025-10-10T00:53:29.9694632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9694857Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9695314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9695461Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9695468Z 2025-10-10T00:53:29.9695646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9696010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9696126Z res = mod(**inputs) 2025-10-10T00:53:29.9696631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9696749Z outputs = self.bert( 2025-10-10T00:53:29.9697211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9697333Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9697788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9697896Z layer_outputs = layer_module( 2025-10-10T00:53:29.9698324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9698438Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9698909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9699031Z return func(*args, **kwargs) 2025-10-10T00:53:29.9699515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9699653Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9700121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9700250Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9700739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9700928Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9701371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9701493Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9701596Z 2025-10-10T00:53:29.9701833Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9702184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9702283Z res = mod(**inputs) 2025-10-10T00:53:29.9702756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9702869Z outputs = self.bert( 2025-10-10T00:53:29.9703329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9703449Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9703913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9704027Z layer_outputs = layer_module( 2025-10-10T00:53:29.9704444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9704589Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9705045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9705169Z return func(*args, **kwargs) 2025-10-10T00:53:29.9705640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9705775Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9706407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9706540Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9707073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9707291Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9707765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9707957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9708360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9708492Z return self.act(input) 2025-10-10T00:53:29.9708499Z 2025-10-10T00:53:29.9708682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9709073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9709176Z res = mod(**inputs) 2025-10-10T00:53:29.9709629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9709745Z outputs = self.bert( 2025-10-10T00:53:29.9710221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9710354Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9710833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9710949Z layer_outputs = layer_module( 2025-10-10T00:53:29.9711396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9711538Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9711992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9712109Z return func(*args, **kwargs) 2025-10-10T00:53:29.9712618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9712757Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9713403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9713549Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9714092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9714322Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9714787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9714926Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9714942Z 2025-10-10T00:53:29.9715133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9715519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9715640Z res = mod(**inputs) 2025-10-10T00:53:29.9716142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9716270Z outputs = self.bert( 2025-10-10T00:53:29.9716770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9716887Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9717392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9717514Z layer_outputs = layer_module( 2025-10-10T00:53:29.9717924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9718052Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9718543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9718688Z return func(*args, **kwargs) 2025-10-10T00:53:29.9719154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9719310Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9719797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9719909Z return func(*args, **kwargs) 2025-10-10T00:53:29.9720411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9720524Z self_outputs = self.self( 2025-10-10T00:53:29.9721004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9721120Z return func(*args, **kwargs) 2025-10-10T00:53:29.9721608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-10-10T00:53:29.9721988Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T00:53:29.9721995Z 2025-10-10T00:53:29.9722170Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9722643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9722781Z res = mod(**inputs) 2025-10-10T00:53:29.9723276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9723382Z outputs = self.bert( 2025-10-10T00:53:29.9723832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9723968Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9724562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9724751Z layer_outputs = layer_module( 2025-10-10T00:53:29.9725158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9725291Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9725740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9725852Z return func(*args, **kwargs) 2025-10-10T00:53:29.9726330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9726474Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9726938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9727054Z return func(*args, **kwargs) 2025-10-10T00:53:29.9727512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9727638Z self_outputs = self.self( 2025-10-10T00:53:29.9728090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9728214Z return func(*args, **kwargs) 2025-10-10T00:53:29.9728694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-10-10T00:53:29.9728806Z self.key(current_states) 2025-10-10T00:53:29.9728823Z 2025-10-10T00:53:29.9729023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9729374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9729486Z res = mod(**inputs) 2025-10-10T00:53:29.9729933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9730057Z outputs = self.bert( 2025-10-10T00:53:29.9730534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9730658Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9731165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9731292Z layer_outputs = layer_module( 2025-10-10T00:53:29.9731949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9732086Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9732549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9732680Z return func(*args, **kwargs) 2025-10-10T00:53:29.9733180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9733347Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9733828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9733949Z return func(*args, **kwargs) 2025-10-10T00:53:29.9734445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9734569Z self_outputs = self.self( 2025-10-10T00:53:29.9735050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9735163Z return func(*args, **kwargs) 2025-10-10T00:53:29.9735641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-10-10T00:53:29.9735761Z self.value(current_states) 2025-10-10T00:53:29.9736059Z 2025-10-10T00:53:29.9736209Z cudagraph partition due to non gpu ops 2025-10-10T00:53:29.9736418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9736799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9736910Z res = mod(**inputs) 2025-10-10T00:53:29.9737404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9737509Z outputs = self.bert( 2025-10-10T00:53:29.9738009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9738135Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9738615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9738737Z layer_outputs = layer_module( 2025-10-10T00:53:29.9739187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9739332Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9739796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9739914Z return func(*args, **kwargs) 2025-10-10T00:53:29.9740350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9740504Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9740972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9741088Z return func(*args, **kwargs) 2025-10-10T00:53:29.9741584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-10-10T00:53:29.9741708Z self_outputs = self.self( 2025-10-10T00:53:29.9742178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9742281Z return func(*args, **kwargs) 2025-10-10T00:53:29.9742764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-10-10T00:53:29.9743028Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:53:29.9743037Z 2025-10-10T00:53:29.9743245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9743642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9743751Z res = mod(**inputs) 2025-10-10T00:53:29.9744262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9744391Z outputs = self.bert( 2025-10-10T00:53:29.9744884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9745009Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9745484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9745625Z layer_outputs = layer_module( 2025-10-10T00:53:29.9746188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9746337Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9746820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9746943Z return func(*args, **kwargs) 2025-10-10T00:53:29.9747411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-10-10T00:53:29.9747718Z self_attention_outputs = self.attention( 2025-10-10T00:53:29.9748172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9748299Z return func(*args, **kwargs) 2025-10-10T00:53:29.9748789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-10-10T00:53:29.9749045Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:53:29.9749519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-10-10T00:53:29.9749663Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9749671Z 2025-10-10T00:53:29.9749857Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9750254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9750388Z res = mod(**inputs) 2025-10-10T00:53:29.9750890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9751008Z outputs = self.bert( 2025-10-10T00:53:29.9751489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9751621Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9752129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9752256Z layer_outputs = layer_module( 2025-10-10T00:53:29.9752688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9752815Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9753287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9753402Z return func(*args, **kwargs) 2025-10-10T00:53:29.9753869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9754033Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9754520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9754661Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9755158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9755355Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9755817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-10-10T00:53:29.9755966Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9755983Z 2025-10-10T00:53:29.9756168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9756501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9756591Z res = mod(**inputs) 2025-10-10T00:53:29.9757004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9757098Z outputs = self.bert( 2025-10-10T00:53:29.9757525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9757635Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9758099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9758213Z layer_outputs = layer_module( 2025-10-10T00:53:29.9758711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9758902Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9759338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9759460Z return func(*args, **kwargs) 2025-10-10T00:53:29.9759920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9760064Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9760563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9760689Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9761183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-10-10T00:53:29.9761388Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:53:29.9761824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-10-10T00:53:29.9762016Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:53:29.9762405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:53:29.9762528Z return self.act(input) 2025-10-10T00:53:29.9762535Z 2025-10-10T00:53:29.9762718Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9763092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9763200Z res = mod(**inputs) 2025-10-10T00:53:29.9763678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-10-10T00:53:29.9763795Z outputs = self.bert( 2025-10-10T00:53:29.9764244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-10-10T00:53:29.9764370Z encoder_outputs = self.encoder( 2025-10-10T00:53:29.9764803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-10-10T00:53:29.9764913Z layer_outputs = layer_module( 2025-10-10T00:53:29.9765321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:53:29.9765450Z return super().__call__(*args, **kwargs) 2025-10-10T00:53:29.9765899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:53:29.9766011Z return func(*args, **kwargs) 2025-10-10T00:53:29.9766485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-10-10T00:53:29.9766650Z layer_output = apply_chunking_to_forward( 2025-10-10T00:53:29.9767164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:53:29.9767305Z return forward_fn(*input_tensors) 2025-10-10T00:53:29.9767798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-10-10T00:53:29.9768027Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:53:29.9768478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-10-10T00:53:29.9768620Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9768629Z 2025-10-10T00:53:29.9768821Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9769208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9769475Z res = mod(**inputs) 2025-10-10T00:53:29.9769942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1294, in forward 2025-10-10T00:53:29.9770104Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:53:29.9770591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 751, in forward 2025-10-10T00:53:29.9770803Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:53:29.9771234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 740, in forward 2025-10-10T00:53:29.9771381Z hidden_states = self.transform(hidden_states) 2025-10-10T00:53:29.9771844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 716, in forward 2025-10-10T00:53:29.9771979Z hidden_states = self.dense(hidden_states) 2025-10-10T00:53:29.9771993Z 2025-10-10T00:53:29.9772174Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9772522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9772623Z res = mod(**inputs) 2025-10-10T00:53:29.9773057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1294, in forward 2025-10-10T00:53:29.9773205Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:53:29.9773628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 751, in forward 2025-10-10T00:53:29.9773803Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:53:29.9774259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 741, in forward 2025-10-10T00:53:29.9774412Z hidden_states = self.decoder(hidden_states) 2025-10-10T00:53:29.9774420Z 2025-10-10T00:53:29.9774621Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:53:29.9774975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:53:29.9775078Z res = mod(**inputs) 2025-10-10T00:53:29.9775534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1299, in forward 2025-10-10T00:53:29.9775868Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:53:29.9775875Z 2025-10-10T00:53:40.3998853Z Compilation time (from dynamo_timed): 17.236731403 2025-10-10T00:53:40.4072304Z pass 2025-10-10T00:53:40.4072939Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:53:40.4075608Z TIMING: _recursive_pre_grad_passes:0.0073 _recursive_joint_graph_passes:0.40625 _recursive_post_grad_passes:0.07406 async_compile.wait:0.69957 code_gen:9.75351 inductor_compile:11.08229 backend_compile:14.37461 gc:0.00058 entire_frame_compile:17.23673 total_wall_time:17.23673 2025-10-10T00:53:40.4076661Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:7255 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:1946 2025-10-10T00:53:40.4077218Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-10-10T00:53:43.1167716Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:53:43.1168627Z import pynvml # type: ignore[import] 2025-10-10T00:53:46.5630282Z 2025-10-10T00:54:06.1605008Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:54:06.1608976Z loading model: 0it [00:19, ?it/s] 2025-10-10T00:54:06.1632473Z cpu eval BlenderbotForCausalLM 2025-10-10T00:54:06.1825646Z Compilation time (from dynamo_timed): 0 2025-10-10T00:54:06.1827057Z pass_due_to_skip 2025-10-10T00:54:06.1827449Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:06.1827819Z TIMING: total_wall_time:0 2025-10-10T00:54:06.1828028Z STATS: call_* op count: 0 2025-10-10T00:54:06.1828495Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-10-10T00:54:08.3892933Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:54:08.3893907Z import pynvml # type: ignore[import] 2025-10-10T00:54:11.8803568Z 2025-10-10T00:54:20.8235036Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:54:20.8243534Z loading model: 0it [00:08, ?it/s] 2025-10-10T00:54:20.8267482Z cpu eval DebertaV2ForMaskedLM 2025-10-10T00:54:20.8464487Z Compilation time (from dynamo_timed): 0 2025-10-10T00:54:20.8464793Z pass_due_to_skip 2025-10-10T00:54:20.8465129Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:20.8465478Z TIMING: total_wall_time:0 2025-10-10T00:54:20.8465686Z STATS: call_* op count: 0 2025-10-10T00:54:20.8465968Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-10-10T00:54:22.8769737Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:54:22.8770691Z import pynvml # type: ignore[import] 2025-10-10T00:54:26.3744167Z 2025-10-10T00:54:27.1204731Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:54:27.1205031Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:54:27.1211227Z cpu eval DistilBertForMaskedLM 2025-10-10T00:54:27.2811334Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:27.3415500Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:27.3969727Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:32.5877746Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5881240Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5882194Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5882501Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5886973Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5887292Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5891571Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5892162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5892574Z res = mod(**inputs) 2025-10-10T00:54:32.5893045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5893509Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5893963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5894495Z return self.transformer( 2025-10-10T00:54:32.5894957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5895417Z layer_outputs = layer_module( 2025-10-10T00:54:32.5895810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5896268Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5897198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5897665Z sa_output = self.attention( 2025-10-10T00:54:32.5898100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.5898601Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.5898848Z 2025-10-10T00:54:32.5898973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5899378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5899739Z res = mod(**inputs) 2025-10-10T00:54:32.5900178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5900660Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5901146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5901604Z return self.transformer( 2025-10-10T00:54:32.5902043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5902492Z layer_outputs = layer_module( 2025-10-10T00:54:32.5902879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5903290Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5903755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5904211Z sa_output = self.attention( 2025-10-10T00:54:32.5904650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.5905162Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.5905365Z 2025-10-10T00:54:32.5905485Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5905899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5906493Z res = mod(**inputs) 2025-10-10T00:54:32.5906922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5907394Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5907835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5908279Z return self.transformer( 2025-10-10T00:54:32.5908712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5909155Z layer_outputs = layer_module( 2025-10-10T00:54:32.5909535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5909943Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5910394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5910830Z sa_output = self.attention( 2025-10-10T00:54:32.5911261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.5911760Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.5911954Z 2025-10-10T00:54:32.5912054Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5912325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5912823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5913243Z res = mod(**inputs) 2025-10-10T00:54:32.5913685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5914154Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5914619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5915083Z return self.transformer( 2025-10-10T00:54:32.5915523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5916119Z layer_outputs = layer_module( 2025-10-10T00:54:32.5916501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5916894Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5917356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5917806Z sa_output = self.attention( 2025-10-10T00:54:32.5918244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.5918760Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.5918966Z 2025-10-10T00:54:32.5919083Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5919477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5919834Z res = mod(**inputs) 2025-10-10T00:54:32.5920258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5920719Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5921164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5921611Z return self.transformer( 2025-10-10T00:54:32.5922046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5922496Z layer_outputs = layer_module( 2025-10-10T00:54:32.5922878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5923274Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5923740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5924206Z sa_output = self.attention( 2025-10-10T00:54:32.5924644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.5925099Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.5925259Z 2025-10-10T00:54:32.5925374Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5925766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5926122Z res = mod(**inputs) 2025-10-10T00:54:32.5926542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5926986Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5927436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5927885Z return self.transformer( 2025-10-10T00:54:32.5928319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5928871Z layer_outputs = layer_module( 2025-10-10T00:54:32.5929246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5929640Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5930090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.5930576Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.5931052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.5931820Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.5932404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.5932830Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.5933262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.5933679Z x = self.lin1(input) 2025-10-10T00:54:32.5933802Z 2025-10-10T00:54:32.5933913Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5934290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5934625Z res = mod(**inputs) 2025-10-10T00:54:32.5935029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5935449Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5935875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5936309Z return self.transformer( 2025-10-10T00:54:32.5936758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5937178Z layer_outputs = layer_module( 2025-10-10T00:54:32.5937526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5937898Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5938324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.5938785Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.5939235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.5939786Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.5940323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.5940731Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.5941157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.5941573Z x = self.activation(x) 2025-10-10T00:54:32.5941914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.5942267Z return self.act(input) 2025-10-10T00:54:32.5942379Z 2025-10-10T00:54:32.5942494Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5942866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5943190Z res = mod(**inputs) 2025-10-10T00:54:32.5943695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5944185Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5944632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5945079Z return self.transformer( 2025-10-10T00:54:32.5945501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5945949Z layer_outputs = layer_module( 2025-10-10T00:54:32.5946415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5946831Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5947304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.5947773Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.5948238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.5948791Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.5949336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.5949718Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.5950124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.5950523Z x = self.lin2(x) 2025-10-10T00:54:32.5950621Z 2025-10-10T00:54:32.5950732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5951090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5951403Z res = mod(**inputs) 2025-10-10T00:54:32.5951783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5952186Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5952585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5952984Z return self.transformer( 2025-10-10T00:54:32.5953364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5953762Z layer_outputs = layer_module( 2025-10-10T00:54:32.5954102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5954460Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5954857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5955254Z sa_output = self.attention( 2025-10-10T00:54:32.5955640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.5956088Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.5956262Z 2025-10-10T00:54:32.5956370Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5956713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5957026Z res = mod(**inputs) 2025-10-10T00:54:32.5957400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5957800Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5958233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5958651Z return self.transformer( 2025-10-10T00:54:32.5959038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5959436Z layer_outputs = layer_module( 2025-10-10T00:54:32.5959778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5960134Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5960531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5960928Z sa_output = self.attention( 2025-10-10T00:54:32.5961317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.5961763Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.5961934Z 2025-10-10T00:54:32.5962035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5962382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5962695Z res = mod(**inputs) 2025-10-10T00:54:32.5963068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5963470Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5963858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5964267Z return self.transformer( 2025-10-10T00:54:32.5964664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5965077Z layer_outputs = layer_module( 2025-10-10T00:54:32.5965423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5965800Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5966214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5966621Z sa_output = self.attention( 2025-10-10T00:54:32.5967021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.5967472Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.5967654Z 2025-10-10T00:54:32.5967739Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.5967980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5968345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5968664Z res = mod(**inputs) 2025-10-10T00:54:32.5969031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5969432Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5969837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5970238Z return self.transformer( 2025-10-10T00:54:32.5970628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5971019Z layer_outputs = layer_module( 2025-10-10T00:54:32.5971379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5971735Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5972201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5972595Z sa_output = self.attention( 2025-10-10T00:54:32.5972984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.5973441Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.5973623Z 2025-10-10T00:54:32.5973732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5974091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5974398Z res = mod(**inputs) 2025-10-10T00:54:32.5974774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5975171Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5975571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5975970Z return self.transformer( 2025-10-10T00:54:32.5976351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5976759Z layer_outputs = layer_module( 2025-10-10T00:54:32.5977110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5977476Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5977885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.5978300Z sa_output = self.attention( 2025-10-10T00:54:32.5978692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.5979108Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.5979242Z 2025-10-10T00:54:32.5979351Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5979691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5980004Z res = mod(**inputs) 2025-10-10T00:54:32.5980378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5980794Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5981201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5981602Z return self.transformer( 2025-10-10T00:54:32.5981994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5982409Z layer_outputs = layer_module( 2025-10-10T00:54:32.5982757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5987584Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5988014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.5988466Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.5988916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.5989448Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.5989951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.5990351Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.5990851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.5991308Z x = self.lin1(input) 2025-10-10T00:54:32.5991420Z 2025-10-10T00:54:32.5991529Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.5991894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.5992214Z res = mod(**inputs) 2025-10-10T00:54:32.5992592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.5993001Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.5993398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.5993803Z return self.transformer( 2025-10-10T00:54:32.5994201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.5994607Z layer_outputs = layer_module( 2025-10-10T00:54:32.5994949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.5995325Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.5995767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.5996248Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.5996737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.5997259Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.5997767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.5998160Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.5998571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.5998990Z x = self.activation(x) 2025-10-10T00:54:32.6011005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.6011538Z return self.act(input) 2025-10-10T00:54:32.6011682Z 2025-10-10T00:54:32.6011808Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6012208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6012569Z res = mod(**inputs) 2025-10-10T00:54:32.6013010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6013467Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6013905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6014463Z return self.transformer( 2025-10-10T00:54:32.6014883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6015300Z layer_outputs = layer_module( 2025-10-10T00:54:32.6015683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6016093Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6016550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6017043Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6017625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6018179Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6018711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6019124Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6019553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.6019963Z x = self.lin2(x) 2025-10-10T00:54:32.6020078Z 2025-10-10T00:54:32.6020190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6020572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6020928Z res = mod(**inputs) 2025-10-10T00:54:32.6021342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6021792Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6022237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6022678Z return self.transformer( 2025-10-10T00:54:32.6023109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6023545Z layer_outputs = layer_module( 2025-10-10T00:54:32.6023929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6024328Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6024781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6025226Z sa_output = self.attention( 2025-10-10T00:54:32.6025655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.6026257Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.6026470Z 2025-10-10T00:54:32.6026591Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6027000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6027361Z res = mod(**inputs) 2025-10-10T00:54:32.6027796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6028271Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6028747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6029180Z return self.transformer( 2025-10-10T00:54:32.6029594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6030072Z layer_outputs = layer_module( 2025-10-10T00:54:32.6030432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6030813Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6031247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6031813Z sa_output = self.attention( 2025-10-10T00:54:32.6032230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.6032786Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6032999Z 2025-10-10T00:54:32.6033116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6033486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6033819Z res = mod(**inputs) 2025-10-10T00:54:32.6034225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6034660Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6035089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6035511Z return self.transformer( 2025-10-10T00:54:32.6035931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6036361Z layer_outputs = layer_module( 2025-10-10T00:54:32.6036730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6037113Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6037544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6037974Z sa_output = self.attention( 2025-10-10T00:54:32.6038391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.6038869Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6039056Z 2025-10-10T00:54:32.6039153Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.6039398Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6039766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6040104Z res = mod(**inputs) 2025-10-10T00:54:32.6040507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6040930Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6041356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6041786Z return self.transformer( 2025-10-10T00:54:32.6042197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6042622Z layer_outputs = layer_module( 2025-10-10T00:54:32.6042977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6043353Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6043788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6044217Z sa_output = self.attention( 2025-10-10T00:54:32.6044638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.6045156Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.6045352Z 2025-10-10T00:54:32.6045456Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6045816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6046142Z res = mod(**inputs) 2025-10-10T00:54:32.6046526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6046939Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6047375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6047801Z return self.transformer( 2025-10-10T00:54:32.6048205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6048619Z layer_outputs = layer_module( 2025-10-10T00:54:32.6048977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6049345Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6049764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6050177Z sa_output = self.attention( 2025-10-10T00:54:32.6050572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.6051001Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.6051150Z 2025-10-10T00:54:32.6051255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6051618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6051940Z res = mod(**inputs) 2025-10-10T00:54:32.6052332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6052749Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6053159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6053573Z return self.transformer( 2025-10-10T00:54:32.6053967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6054382Z layer_outputs = layer_module( 2025-10-10T00:54:32.6054738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6055106Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6055527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6055975Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6056431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6056974Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6057495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6057899Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6058310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.6058724Z x = self.lin1(input) 2025-10-10T00:54:32.6058863Z 2025-10-10T00:54:32.6058968Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6059327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6059700Z res = mod(**inputs) 2025-10-10T00:54:32.6060080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6060480Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6060880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6061289Z return self.transformer( 2025-10-10T00:54:32.6061709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6062138Z layer_outputs = layer_module( 2025-10-10T00:54:32.6062496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6062872Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6063327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6063818Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6064317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6064902Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6065464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6065900Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6066447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.6066923Z x = self.activation(x) 2025-10-10T00:54:32.6067300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.6067650Z return self.act(input) 2025-10-10T00:54:32.6067764Z 2025-10-10T00:54:32.6067870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6068241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6068570Z res = mod(**inputs) 2025-10-10T00:54:32.6068963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6069381Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6069780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6070194Z return self.transformer( 2025-10-10T00:54:32.6070586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6070991Z layer_outputs = layer_module( 2025-10-10T00:54:32.6071337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6071692Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6072109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6072553Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6073005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6073534Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6074060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6074477Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6074911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.6075335Z x = self.lin2(x) 2025-10-10T00:54:32.6075444Z 2025-10-10T00:54:32.6075567Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6075951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6076291Z res = mod(**inputs) 2025-10-10T00:54:32.6076726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6077137Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6077538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6077958Z return self.transformer( 2025-10-10T00:54:32.6078365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6078788Z layer_outputs = layer_module( 2025-10-10T00:54:32.6079145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6079509Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6079942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6080362Z sa_output = self.attention( 2025-10-10T00:54:32.6080774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.6081252Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.6081442Z 2025-10-10T00:54:32.6081549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6081920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6082256Z res = mod(**inputs) 2025-10-10T00:54:32.6082657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6083076Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6083498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6083917Z return self.transformer( 2025-10-10T00:54:32.6084330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6084752Z layer_outputs = layer_module( 2025-10-10T00:54:32.6085112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6085481Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6085945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6086382Z sa_output = self.attention( 2025-10-10T00:54:32.6086790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.6087259Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6087440Z 2025-10-10T00:54:32.6087551Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6087917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6088265Z res = mod(**inputs) 2025-10-10T00:54:32.6088664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6089086Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6089498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6089914Z return self.transformer( 2025-10-10T00:54:32.6090299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6090698Z layer_outputs = layer_module( 2025-10-10T00:54:32.6091086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6091447Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6091860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6092261Z sa_output = self.attention( 2025-10-10T00:54:32.6092653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.6093101Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6093289Z 2025-10-10T00:54:32.6093373Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.6093621Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6093982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6094316Z res = mod(**inputs) 2025-10-10T00:54:32.6094717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6095152Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6095610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6096071Z return self.transformer( 2025-10-10T00:54:32.6096506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6096964Z layer_outputs = layer_module( 2025-10-10T00:54:32.6097346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6097794Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6098227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6098647Z sa_output = self.attention( 2025-10-10T00:54:32.6099063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.6099556Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.6099749Z 2025-10-10T00:54:32.6099866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6100243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6100593Z res = mod(**inputs) 2025-10-10T00:54:32.6101026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6101485Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6101938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6102377Z return self.transformer( 2025-10-10T00:54:32.6102819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6103289Z layer_outputs = layer_module( 2025-10-10T00:54:32.6103668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6104065Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6104514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6104969Z sa_output = self.attention( 2025-10-10T00:54:32.6105411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.6105878Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.6106159Z 2025-10-10T00:54:32.6106293Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6106678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6107033Z res = mod(**inputs) 2025-10-10T00:54:32.6107465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6107925Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6108373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6108830Z return self.transformer( 2025-10-10T00:54:32.6109272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6109728Z layer_outputs = layer_module( 2025-10-10T00:54:32.6110113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6110505Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6110961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6111454Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6111938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6112527Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6113087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6113535Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6114005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.6114470Z x = self.lin1(input) 2025-10-10T00:54:32.6114597Z 2025-10-10T00:54:32.6114719Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6115116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6115478Z res = mod(**inputs) 2025-10-10T00:54:32.6115912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6116375Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6116825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6117288Z return self.transformer( 2025-10-10T00:54:32.6117735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6118194Z layer_outputs = layer_module( 2025-10-10T00:54:32.6118573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6118998Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6119447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6119945Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6120439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6121038Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6121633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6122100Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6122570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.6123032Z x = self.activation(x) 2025-10-10T00:54:32.6123408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.6123782Z return self.act(input) 2025-10-10T00:54:32.6123912Z 2025-10-10T00:54:32.6124027Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6124431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6124792Z res = mod(**inputs) 2025-10-10T00:54:32.6125218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6125683Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6126137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6126591Z return self.transformer( 2025-10-10T00:54:32.6127033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6127481Z layer_outputs = layer_module( 2025-10-10T00:54:32.6127878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6128298Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6128764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6129263Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6129757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6130357Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6130929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6131374Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6132042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.6132492Z x = self.lin2(x) 2025-10-10T00:54:32.6132616Z 2025-10-10T00:54:32.6132733Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6133140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6133511Z res = mod(**inputs) 2025-10-10T00:54:32.6133943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6134412Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6134942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6135395Z return self.transformer( 2025-10-10T00:54:32.6135845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6136294Z layer_outputs = layer_module( 2025-10-10T00:54:32.6136660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6137046Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6137480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6137994Z sa_output = self.attention( 2025-10-10T00:54:32.6138401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.6138875Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.6139066Z 2025-10-10T00:54:32.6139173Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6139537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6139868Z res = mod(**inputs) 2025-10-10T00:54:32.6140253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6140672Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6141081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6141499Z return self.transformer( 2025-10-10T00:54:32.6141895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6142319Z layer_outputs = layer_module( 2025-10-10T00:54:32.6142692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6143087Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6143529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6143961Z sa_output = self.attention( 2025-10-10T00:54:32.6144389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.6144879Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6145072Z 2025-10-10T00:54:32.6145193Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6145576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6145917Z res = mod(**inputs) 2025-10-10T00:54:32.6146414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6146898Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6147355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6147798Z return self.transformer( 2025-10-10T00:54:32.6148208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6148630Z layer_outputs = layer_module( 2025-10-10T00:54:32.6148991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6149377Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6149804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6150290Z sa_output = self.attention( 2025-10-10T00:54:32.6150688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.6151173Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6151353Z 2025-10-10T00:54:32.6151445Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.6151687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6152042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6152367Z res = mod(**inputs) 2025-10-10T00:54:32.6152828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6153247Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6153660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6154079Z return self.transformer( 2025-10-10T00:54:32.6154489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6154896Z layer_outputs = layer_module( 2025-10-10T00:54:32.6155245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6155602Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6156021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6156427Z sa_output = self.attention( 2025-10-10T00:54:32.6156820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.6157283Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.6157467Z 2025-10-10T00:54:32.6157569Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6157930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6158247Z res = mod(**inputs) 2025-10-10T00:54:32.6158628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6159026Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6159434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6159846Z return self.transformer( 2025-10-10T00:54:32.6160239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6160644Z layer_outputs = layer_module( 2025-10-10T00:54:32.6160983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6161343Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6161755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6162157Z sa_output = self.attention( 2025-10-10T00:54:32.6162549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.6162959Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.6163108Z 2025-10-10T00:54:32.6163212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6163570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6163921Z res = mod(**inputs) 2025-10-10T00:54:32.6164294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6164708Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6165113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6165521Z return self.transformer( 2025-10-10T00:54:32.6165919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6166324Z layer_outputs = layer_module( 2025-10-10T00:54:32.6166704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6167083Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6167500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6167950Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6168389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6168925Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6169440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6169847Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6170257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.6170657Z x = self.lin1(input) 2025-10-10T00:54:32.6170776Z 2025-10-10T00:54:32.6170882Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6171237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6171556Z res = mod(**inputs) 2025-10-10T00:54:32.6171943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6172365Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6172796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6173207Z return self.transformer( 2025-10-10T00:54:32.6173606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6174010Z layer_outputs = layer_module( 2025-10-10T00:54:32.6174359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6174727Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6175141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6175629Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6176100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6176648Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6177179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6177567Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6177969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.6178380Z x = self.activation(x) 2025-10-10T00:54:32.6178697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.6179035Z return self.act(input) 2025-10-10T00:54:32.6179145Z 2025-10-10T00:54:32.6179256Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6179612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6179928Z res = mod(**inputs) 2025-10-10T00:54:32.6180314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6180728Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6181191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6181603Z return self.transformer( 2025-10-10T00:54:32.6182014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6182430Z layer_outputs = layer_module( 2025-10-10T00:54:32.6182808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6183234Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6183729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6184188Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6184672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6185249Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6185800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6186305Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6186777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.6187245Z x = self.lin2(x) 2025-10-10T00:54:32.6187362Z 2025-10-10T00:54:32.6187489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6187901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6188263Z res = mod(**inputs) 2025-10-10T00:54:32.6188695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6189130Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6189557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6189979Z return self.transformer( 2025-10-10T00:54:32.6190394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6190820Z layer_outputs = layer_module( 2025-10-10T00:54:32.6191185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6191565Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6191990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6192426Z sa_output = self.attention( 2025-10-10T00:54:32.6192845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-10-10T00:54:32.6193361Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-10-10T00:54:32.6193553Z 2025-10-10T00:54:32.6193674Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6194053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6194386Z res = mod(**inputs) 2025-10-10T00:54:32.6194778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6195196Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6195604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6196071Z return self.transformer( 2025-10-10T00:54:32.6196479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6196897Z layer_outputs = layer_module( 2025-10-10T00:54:32.6197250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6197618Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6198050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6198445Z sa_output = self.attention( 2025-10-10T00:54:32.6198843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-10-10T00:54:32.6199309Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6199489Z 2025-10-10T00:54:32.6199598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6199966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6200294Z res = mod(**inputs) 2025-10-10T00:54:32.6200693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6201101Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6201502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6201904Z return self.transformer( 2025-10-10T00:54:32.6202300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6202717Z layer_outputs = layer_module( 2025-10-10T00:54:32.6203071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6203451Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6203879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6204305Z sa_output = self.attention( 2025-10-10T00:54:32.6204712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-10-10T00:54:32.6205172Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-10-10T00:54:32.6205372Z 2025-10-10T00:54:32.6205456Z cudagraph partition due to non gpu ops 2025-10-10T00:54:32.6205697Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6206055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6206382Z res = mod(**inputs) 2025-10-10T00:54:32.6206775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6207196Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6207636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6208061Z return self.transformer( 2025-10-10T00:54:32.6208457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6208884Z layer_outputs = layer_module( 2025-10-10T00:54:32.6209245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6209620Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6210048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6210522Z sa_output = self.attention( 2025-10-10T00:54:32.6210920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-10-10T00:54:32.6211389Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:32.6211576Z 2025-10-10T00:54:32.6211686Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6212044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6212369Z res = mod(**inputs) 2025-10-10T00:54:32.6212769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6213190Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6213605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6214017Z return self.transformer( 2025-10-10T00:54:32.6214424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6214839Z layer_outputs = layer_module( 2025-10-10T00:54:32.6215193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6215560Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6215982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-10-10T00:54:32.6216397Z sa_output = self.attention( 2025-10-10T00:54:32.6216806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-10-10T00:54:32.6217241Z attn_output = self.out_lin(attn_output) 2025-10-10T00:54:32.6217383Z 2025-10-10T00:54:32.6217498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6217858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6218188Z res = mod(**inputs) 2025-10-10T00:54:32.6218581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6218998Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6219407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6219856Z return self.transformer( 2025-10-10T00:54:32.6220294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6220744Z layer_outputs = layer_module( 2025-10-10T00:54:32.6221119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6221515Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6221973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6222497Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6222978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6223563Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6224110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6224551Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6225036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-10-10T00:54:32.6225523Z x = self.lin1(input) 2025-10-10T00:54:32.6225645Z 2025-10-10T00:54:32.6225771Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6226239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6226614Z res = mod(**inputs) 2025-10-10T00:54:32.6227044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6227502Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6227943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6228404Z return self.transformer( 2025-10-10T00:54:32.6228844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6229294Z layer_outputs = layer_module( 2025-10-10T00:54:32.6229653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6230024Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6230469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6230965Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6231418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6232164Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6232691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6233109Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6233536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-10-10T00:54:32.6233959Z x = self.activation(x) 2025-10-10T00:54:32.6234297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:54:32.6234650Z return self.act(input) 2025-10-10T00:54:32.6234765Z 2025-10-10T00:54:32.6234870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6235220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6235550Z res = mod(**inputs) 2025-10-10T00:54:32.6235944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-10-10T00:54:32.6236399Z dlbrt_output = self.distilbert( 2025-10-10T00:54:32.6236843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-10-10T00:54:32.6237288Z return self.transformer( 2025-10-10T00:54:32.6237764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-10-10T00:54:32.6238186Z layer_outputs = layer_module( 2025-10-10T00:54:32.6238537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:32.6238909Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:32.6239332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-10-10T00:54:32.6239793Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-10-10T00:54:32.6240311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-10-10T00:54:32.6240874Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-10-10T00:54:32.6241402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:54:32.6241811Z return forward_fn(*input_tensors) 2025-10-10T00:54:32.6242234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-10-10T00:54:32.6242645Z x = self.lin2(x) 2025-10-10T00:54:32.6242756Z 2025-10-10T00:54:32.6242862Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6243249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6243604Z res = mod(**inputs) 2025-10-10T00:54:32.6244019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-10-10T00:54:32.6244561Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-10-10T00:54:32.6244805Z 2025-10-10T00:54:32.6244916Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6245294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6245612Z res = mod(**inputs) 2025-10-10T00:54:32.6245990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 829, in forward 2025-10-10T00:54:32.6246518Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-10-10T00:54:32.6246771Z 2025-10-10T00:54:32.6246875Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:32.6247243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:54:32.6247564Z res = mod(**inputs) 2025-10-10T00:54:32.6247952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 833, in forward 2025-10-10T00:54:32.6248506Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-10-10T00:54:32.6248767Z 2025-10-10T00:54:41.8062462Z Compilation time (from dynamo_timed): 13.273572722 2025-10-10T00:54:41.8062776Z pass 2025-10-10T00:54:41.8063096Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:41.8064161Z TIMING: _recursive_pre_grad_passes:0.00529 _recursive_joint_graph_passes:0.2696 _recursive_post_grad_passes:0.04849 async_compile.wait:0.72406 code_gen:8.92381 inductor_compile:9.96866 backend_compile:11.81354 gc:0.00126 entire_frame_compile:13.27357 total_wall_time:13.27357 2025-10-10T00:54:41.8065363Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:4036 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:1039 2025-10-10T00:54:41.8065928Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-10-10T00:54:44.4498386Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:54:44.4499251Z import pynvml # type: ignore[import] 2025-10-10T00:54:47.9767884Z 2025-10-10T00:54:49.0566122Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-10-10T00:54:49.0566905Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-10-10T00:54:49.0825069Z 2025-10-10T00:54:49.0825771Z loading model: 0it [00:01, ?it/s] 2025-10-10T00:54:49.0837434Z cpu eval DistillGPT2 2025-10-10T00:54:49.5161458Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:49.7090007Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:49.9018778Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:54:56.7816139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7816709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7817162Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7817616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:54:56.7818060Z causal_mask = create_causal_mask( 2025-10-10T00:54:56.7818523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:54:56.7819091Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:54:56.7819658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:54:56.7820201Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:54:56.7820713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 654, in find_packed_sequence_indices 2025-10-10T00:54:56.7821308Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-10-10T00:54:56.7821570Z 2025-10-10T00:54:56.7821684Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7821971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7822492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7822937Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7823372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7823795Z outputs = block( 2025-10-10T00:54:56.7824169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7824596Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7825039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7825484Z return func(*args, **kwargs) 2025-10-10T00:54:56.7825911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7826637Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7827081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7827906Z return func(*args, **kwargs) 2025-10-10T00:54:56.7828328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.7828918Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.7829479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7829950Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7830157Z 2025-10-10T00:54:56.7830290Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7830875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7831377Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7832033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:54:56.7832486Z causal_mask = create_causal_mask( 2025-10-10T00:54:56.7832928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:54:56.7833494Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:54:56.7834076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:54:56.7834587Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:54:56.7835061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-10-10T00:54:56.7835554Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-10-10T00:54:56.7835769Z 2025-10-10T00:54:56.7835864Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7836112Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7836542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7836996Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7837391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:54:56.7837782Z causal_mask = create_causal_mask( 2025-10-10T00:54:56.7838168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:54:56.7838681Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:54:56.7839210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:54:56.7839784Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:54:56.7840246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-10-10T00:54:56.7840687Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-10-10T00:54:56.7840863Z 2025-10-10T00:54:56.7840971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7841393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7841799Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7842197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:54:56.7842586Z causal_mask = create_causal_mask( 2025-10-10T00:54:56.7842970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:54:56.7843499Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:54:56.7844016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:54:56.7844466Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:54:56.7844918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-10-10T00:54:56.7845357Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-10-10T00:54:56.7845529Z 2025-10-10T00:54:56.7845614Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7845933Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7846351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7846759Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7847161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7847555Z outputs = block( 2025-10-10T00:54:56.7847904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7848278Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7848695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7849116Z return func(*args, **kwargs) 2025-10-10T00:54:56.7849509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7849942Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7850373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7850764Z return func(*args, **kwargs) 2025-10-10T00:54:56.7851157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.7851733Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.7852212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.7852743Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.7852960Z 2025-10-10T00:54:56.7853070Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7853507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7853917Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7854326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7854736Z outputs = block( 2025-10-10T00:54:56.7855090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7855491Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7855907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7856303Z return func(*args, **kwargs) 2025-10-10T00:54:56.7856706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7857144Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7857570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7857964Z return func(*args, **kwargs) 2025-10-10T00:54:56.7858384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.7858807Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.7859202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7859640Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7859833Z 2025-10-10T00:54:56.7859955Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7860411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7860841Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7861358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7861760Z outputs = block( 2025-10-10T00:54:56.7862106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7862498Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7862908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7863322Z return func(*args, **kwargs) 2025-10-10T00:54:56.7863726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7864190Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7864644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.7865086Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.7865486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7865926Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7866184Z 2025-10-10T00:54:56.7866310Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7866774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7867219Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7867659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7868055Z outputs = block( 2025-10-10T00:54:56.7868411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7868812Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7869225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7869631Z return func(*args, **kwargs) 2025-10-10T00:54:56.7870026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7870477Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7870920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.7871361Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.7871737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.7872242Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.7872511Z 2025-10-10T00:54:56.7872619Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7873047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7873472Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7873860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7874240Z outputs = block( 2025-10-10T00:54:56.7874573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7874948Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7875338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7875711Z return func(*args, **kwargs) 2025-10-10T00:54:56.7876142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7876560Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7876977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.7877381Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.7877746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7878149Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7878331Z 2025-10-10T00:54:56.7878439Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7878861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7879252Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7879651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7880033Z outputs = block( 2025-10-10T00:54:56.7880369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7880742Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7881123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7881555Z return func(*args, **kwargs) 2025-10-10T00:54:56.7881950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7882387Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7882843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7883219Z return func(*args, **kwargs) 2025-10-10T00:54:56.7883594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.7884098Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.7884571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7885004Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7885192Z 2025-10-10T00:54:56.7885290Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7885537Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7885961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7886361Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7886752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7887128Z outputs = block( 2025-10-10T00:54:56.7887489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7887865Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7888254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7888634Z return func(*args, **kwargs) 2025-10-10T00:54:56.7889024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7889470Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7889926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7890368Z return func(*args, **kwargs) 2025-10-10T00:54:56.7890779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.7891193Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.7891651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.7892140Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.7892327Z 2025-10-10T00:54:56.7892434Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7892860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7893262Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7893662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7894060Z outputs = block( 2025-10-10T00:54:56.7894404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7894800Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7895187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7895569Z return func(*args, **kwargs) 2025-10-10T00:54:56.7895947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7896341Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7896734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7897112Z return func(*args, **kwargs) 2025-10-10T00:54:56.7897486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.7897875Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.7898239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7898647Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7898822Z 2025-10-10T00:54:56.7898935Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7899361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7899755Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7900155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7900532Z outputs = block( 2025-10-10T00:54:56.7900868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7901244Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7901624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7902044Z return func(*args, **kwargs) 2025-10-10T00:54:56.7902444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7902893Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7903330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.7903787Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.7904178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7904648Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7904854Z 2025-10-10T00:54:56.7904978Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7905427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7905860Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7906371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7906782Z outputs = block( 2025-10-10T00:54:56.7907137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7907523Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7907934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7908341Z return func(*args, **kwargs) 2025-10-10T00:54:56.7908751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7909188Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7909628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.7910061Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.7910443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.7910930Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.7911179Z 2025-10-10T00:54:56.7911292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7911742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7912171Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7912589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7912990Z outputs = block( 2025-10-10T00:54:56.7913333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7913728Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7914137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7914544Z return func(*args, **kwargs) 2025-10-10T00:54:56.7914940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7915396Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7915834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.7916261Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.7916648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7917087Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7917277Z 2025-10-10T00:54:56.7917389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7917834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7918269Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7918681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7919073Z outputs = block( 2025-10-10T00:54:56.7919476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7919895Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7920315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7920732Z return func(*args, **kwargs) 2025-10-10T00:54:56.7921134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7921572Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7921991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7922391Z return func(*args, **kwargs) 2025-10-10T00:54:56.7922789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.7923337Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.7923852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7924297Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7924487Z 2025-10-10T00:54:56.7924589Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7924853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7925310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7925746Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7926174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7926584Z outputs = block( 2025-10-10T00:54:56.7926937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7927347Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7927765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7928180Z return func(*args, **kwargs) 2025-10-10T00:54:56.7928587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7929020Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7929449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7929859Z return func(*args, **kwargs) 2025-10-10T00:54:56.7930268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.7930708Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.7931207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.7931899Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.7932163Z 2025-10-10T00:54:56.7932289Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7932754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7933186Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7933619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7934040Z outputs = block( 2025-10-10T00:54:56.7934400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7934816Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7935319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7935698Z return func(*args, **kwargs) 2025-10-10T00:54:56.7936081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7936491Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7936883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7937269Z return func(*args, **kwargs) 2025-10-10T00:54:56.7937650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.7938051Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.7938423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7938833Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7939019Z 2025-10-10T00:54:56.7939132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7939565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7939973Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7940374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7940753Z outputs = block( 2025-10-10T00:54:56.7941095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7941475Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7941870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7942260Z return func(*args, **kwargs) 2025-10-10T00:54:56.7942639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7943067Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7943493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.7943897Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.7944278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7944719Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7944926Z 2025-10-10T00:54:56.7945044Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7945496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7945940Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7946427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7946881Z outputs = block( 2025-10-10T00:54:56.7947247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7947656Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7948057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7948429Z return func(*args, **kwargs) 2025-10-10T00:54:56.7948805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7949225Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7949676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.7950084Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.7950447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.7950900Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.7951127Z 2025-10-10T00:54:56.7951231Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7951649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7952030Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7952418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7952784Z outputs = block( 2025-10-10T00:54:56.7953118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7953467Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7953827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7954183Z return func(*args, **kwargs) 2025-10-10T00:54:56.7954540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7954938Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7955329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.7955700Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.7956050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7956435Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7956601Z 2025-10-10T00:54:56.7956707Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7957102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7957489Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7957872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7958240Z outputs = block( 2025-10-10T00:54:56.7958559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7958927Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7959293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7959660Z return func(*args, **kwargs) 2025-10-10T00:54:56.7960020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:54:56.7960438Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:54:56.7960593Z 2025-10-10T00:54:56.7960693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7961108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7961502Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7961897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7962276Z outputs = block( 2025-10-10T00:54:56.7962597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7963014Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7963417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7963801Z return func(*args, **kwargs) 2025-10-10T00:54:56.7964177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7964587Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7964967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7965327Z return func(*args, **kwargs) 2025-10-10T00:54:56.7965687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.7966162Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.7966615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7967006Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7967173Z 2025-10-10T00:54:56.7967263Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.7967501Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7967908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7968299Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7968684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7969057Z outputs = block( 2025-10-10T00:54:56.7969375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7969743Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7970128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7970503Z return func(*args, **kwargs) 2025-10-10T00:54:56.7970873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7971262Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7971648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7972019Z return func(*args, **kwargs) 2025-10-10T00:54:56.7972384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.7972786Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.7973227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.7973710Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.7973899Z 2025-10-10T00:54:56.7974003Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7974463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7974868Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7975257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7975638Z outputs = block( 2025-10-10T00:54:56.7975969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7976351Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7976762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7977194Z return func(*args, **kwargs) 2025-10-10T00:54:56.7977568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.7977968Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.7978356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7978724Z return func(*args, **kwargs) 2025-10-10T00:54:56.7979090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.7979480Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.7979846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7980252Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7980441Z 2025-10-10T00:54:56.7980562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7981019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7981449Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7981869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7982268Z outputs = block( 2025-10-10T00:54:56.7982608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7983002Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7983418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7983818Z return func(*args, **kwargs) 2025-10-10T00:54:56.7984216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7984669Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7985112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.7985540Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.7985932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7986439Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7986635Z 2025-10-10T00:54:56.7986748Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7987252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7987680Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7988098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7988502Z outputs = block( 2025-10-10T00:54:56.7988881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7989341Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7989756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7990156Z return func(*args, **kwargs) 2025-10-10T00:54:56.7990558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7991003Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7991446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.7991918Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.7992299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.7992798Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.7993056Z 2025-10-10T00:54:56.7993173Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7993632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7994064Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.7994483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.7994891Z outputs = block( 2025-10-10T00:54:56.7995227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.7995592Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.7995960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.7996333Z return func(*args, **kwargs) 2025-10-10T00:54:56.7996697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.7997103Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.7997504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.7997889Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.7998248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.7998641Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.7998811Z 2025-10-10T00:54:56.7998925Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.7999335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.7999717Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8000101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8000466Z outputs = block( 2025-10-10T00:54:56.8000786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8001140Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8001518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8001884Z return func(*args, **kwargs) 2025-10-10T00:54:56.8002255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8002647Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8003023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8003403Z return func(*args, **kwargs) 2025-10-10T00:54:56.8003759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.8004235Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.8004676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8005054Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8005230Z 2025-10-10T00:54:56.8005313Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.8005612Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8006017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8006407Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8006796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8007156Z outputs = block( 2025-10-10T00:54:56.8007473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8007829Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8008189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8008550Z return func(*args, **kwargs) 2025-10-10T00:54:56.8008910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8009298Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8009680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8010043Z return func(*args, **kwargs) 2025-10-10T00:54:56.8010409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.8010810Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.8011256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.8011731Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.8011913Z 2025-10-10T00:54:56.8012017Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8012439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8012833Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8013220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8013584Z outputs = block( 2025-10-10T00:54:56.8013906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8014267Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8014645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8015017Z return func(*args, **kwargs) 2025-10-10T00:54:56.8015378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8015785Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8016182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8016587Z return func(*args, **kwargs) 2025-10-10T00:54:56.8016957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.8017343Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.8017703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8018102Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8018272Z 2025-10-10T00:54:56.8018387Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8018801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8019252Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8019638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8020017Z outputs = block( 2025-10-10T00:54:56.8020349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8020716Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8021104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8021514Z return func(*args, **kwargs) 2025-10-10T00:54:56.8021914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8022371Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8022810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.8023237Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.8023629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8024060Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8024257Z 2025-10-10T00:54:56.8024379Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8024833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8025257Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8025678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8026171Z outputs = block( 2025-10-10T00:54:56.8026533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8026948Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8027382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8027791Z return func(*args, **kwargs) 2025-10-10T00:54:56.8028178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8028607Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8029051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.8029471Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.8029853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.8030346Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.8030596Z 2025-10-10T00:54:56.8030709Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8031162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8031766Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8032191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8032601Z outputs = block( 2025-10-10T00:54:56.8032951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8033350Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8033768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8034180Z return func(*args, **kwargs) 2025-10-10T00:54:56.8034673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8035121Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8035575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.8036008Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.8036405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8036827Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8037021Z 2025-10-10T00:54:56.8037137Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8037590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8038021Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8038442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8038840Z outputs = block( 2025-10-10T00:54:56.8039188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8039583Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8039995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8040392Z return func(*args, **kwargs) 2025-10-10T00:54:56.8040791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:54:56.8041236Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:54:56.8041397Z 2025-10-10T00:54:56.8041510Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8041940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8042334Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8042729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8043111Z outputs = block( 2025-10-10T00:54:56.8043442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8043813Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8044188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8044569Z return func(*args, **kwargs) 2025-10-10T00:54:56.8044944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8045356Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8045745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8046182Z return func(*args, **kwargs) 2025-10-10T00:54:56.8046558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:54:56.8047057Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:54:56.8047526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8047922Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8048105Z 2025-10-10T00:54:56.8048190Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.8048413Z cudagraph partition due to non gpu ops 2025-10-10T00:54:56.8048710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8049144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8049546Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8049946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8050328Z outputs = block( 2025-10-10T00:54:56.8050660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8051026Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8051420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8051791Z return func(*args, **kwargs) 2025-10-10T00:54:56.8052162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8052558Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8052945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8053323Z return func(*args, **kwargs) 2025-10-10T00:54:56.8053696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:54:56.8054113Z attn_output, attn_weights = attention_interface( 2025-10-10T00:54:56.8054576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:54:56.8055050Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:54:56.8055244Z 2025-10-10T00:54:56.8055352Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8055786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8056178Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8056565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8056929Z outputs = block( 2025-10-10T00:54:56.8057248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8057618Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8058007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8058408Z return func(*args, **kwargs) 2025-10-10T00:54:56.8058812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:54:56.8059219Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:54:56.8059619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8059998Z return func(*args, **kwargs) 2025-10-10T00:54:56.8060392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:54:56.8060821Z attn_output = self.c_proj(attn_output) 2025-10-10T00:54:56.8061208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8061645Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8061832Z 2025-10-10T00:54:56.8061953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8062396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8062821Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8063295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8063695Z outputs = block( 2025-10-10T00:54:56.8064038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8064441Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8064848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8065247Z return func(*args, **kwargs) 2025-10-10T00:54:56.8065645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8066168Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8066642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:54:56.8067093Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:54:56.8067497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8067899Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8068080Z 2025-10-10T00:54:56.8068183Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8068598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8069047Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8069479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8069894Z outputs = block( 2025-10-10T00:54:56.8070256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8070671Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8071102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8071524Z return func(*args, **kwargs) 2025-10-10T00:54:56.8071935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8072401Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8072861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:54:56.8073302Z hidden_states = self.act(hidden_states) 2025-10-10T00:54:56.8073694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:54:56.8074185Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:54:56.8074454Z 2025-10-10T00:54:56.8074572Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8075032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-10-10T00:54:56.8075504Z transformer_outputs = self.transformer( 2025-10-10T00:54:56.8075934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:54:56.8076354Z outputs = block( 2025-10-10T00:54:56.8076702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:54:56.8077100Z return super().__call__(*args, **kwargs) 2025-10-10T00:54:56.8077507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:54:56.8077898Z return func(*args, **kwargs) 2025-10-10T00:54:56.8078364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:54:56.8078812Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:54:56.8079254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:54:56.8079687Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:54:56.8080068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:54:56.8080498Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:54:56.8080691Z 2025-10-10T00:54:56.8080804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:54:56.8081254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1094, in forward 2025-10-10T00:54:56.8081727Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-10-10T00:54:56.8081912Z 2025-10-10T00:55:06.5803880Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:06.5804554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T00:55:06.5805111Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T00:55:06.5805613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T00:55:06.5806145Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T00:55:06.5806412Z 2025-10-10T00:55:07.6147281Z Compilation time (from dynamo_timed): 16.37583567 2025-10-10T00:55:07.6290538Z pass 2025-10-10T00:55:07.6290951Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:07.6298685Z TIMING: gc:0.00322 entire_frame_compile:16.37584 _recursive_pre_grad_passes:0.00742 _recursive_joint_graph_passes:0.24062 _recursive_post_grad_passes:0.05289 async_compile.wait:1.40999 code_gen:9.83797 inductor_compile:10.91932 backend_compile:12.66415 total_wall_time:16.37584 2025-10-10T00:55:07.6299734Z STATS: call_* op count: 311 | FakeTensorMode.__torch_dispatch__:4821 | FakeTensor.__torch_dispatch__:2291 | ProxyTorchDispatchMode.__torch_dispatch__:903 2025-10-10T00:55:07.6300297Z Dynamo produced 2 graphs covering 311 ops with 2 graph breaks (1 unique) 2025-10-10T00:55:10.1691940Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:55:10.1692806Z import pynvml # type: ignore[import] 2025-10-10T00:55:13.6121310Z 2025-10-10T00:55:13.6138365Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-10-10T00:55:13.6139038Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-10-10T00:55:13.8433458Z 2025-10-10T00:55:13.8436863Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:55:13.8449284Z cpu eval ElectraForCausalLM 2025-10-10T00:55:14.0015587Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:14.0866316Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:14.1712355Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:22.7090337Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7091992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7093087Z res = mod(**inputs) 2025-10-10T00:55:22.7093675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7094235Z outputs = self.electra( 2025-10-10T00:55:22.7095292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 787, in forward 2025-10-10T00:55:22.7095930Z hidden_states = self.embeddings_project(hidden_states) 2025-10-10T00:55:22.7096120Z 2025-10-10T00:55:22.7096242Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7096662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7097032Z res = mod(**inputs) 2025-10-10T00:55:22.7097428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7097879Z outputs = self.electra( 2025-10-10T00:55:22.7098300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7098714Z hidden_states = self.encoder( 2025-10-10T00:55:22.7099124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7099531Z layer_outputs = layer_module( 2025-10-10T00:55:22.7099902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7100294Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7100715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7101116Z return func(*args, **kwargs) 2025-10-10T00:55:22.7101510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7101939Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7102355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7102775Z return func(*args, **kwargs) 2025-10-10T00:55:22.7103183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7103630Z self_outputs = self.self( 2025-10-10T00:55:22.7104033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7104444Z return func(*args, **kwargs) 2025-10-10T00:55:22.7104873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7105283Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7105445Z 2025-10-10T00:55:22.7105569Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7105966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7106668Z res = mod(**inputs) 2025-10-10T00:55:22.7107312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7107749Z outputs = self.electra( 2025-10-10T00:55:22.7108154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7108558Z hidden_states = self.encoder( 2025-10-10T00:55:22.7108959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7109376Z layer_outputs = layer_module( 2025-10-10T00:55:22.7109772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7110281Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7110720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7111154Z return func(*args, **kwargs) 2025-10-10T00:55:22.7111565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7112010Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7112443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7112865Z return func(*args, **kwargs) 2025-10-10T00:55:22.7113283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7113704Z self_outputs = self.self( 2025-10-10T00:55:22.7114104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7114520Z return func(*args, **kwargs) 2025-10-10T00:55:22.7114933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7115362Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7115515Z 2025-10-10T00:55:22.7115632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7116032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7116383Z res = mod(**inputs) 2025-10-10T00:55:22.7116792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7117217Z outputs = self.electra( 2025-10-10T00:55:22.7117634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7118060Z hidden_states = self.encoder( 2025-10-10T00:55:22.7118480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7118916Z layer_outputs = layer_module( 2025-10-10T00:55:22.7119301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7119714Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7120144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7120549Z return func(*args, **kwargs) 2025-10-10T00:55:22.7121018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7121463Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7121881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7122282Z return func(*args, **kwargs) 2025-10-10T00:55:22.7122729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7123143Z self_outputs = self.self( 2025-10-10T00:55:22.7123536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7123936Z return func(*args, **kwargs) 2025-10-10T00:55:22.7124347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7124784Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7124930Z 2025-10-10T00:55:22.7125022Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7125261Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7125588Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7125983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7126331Z res = mod(**inputs) 2025-10-10T00:55:22.7126741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7127169Z outputs = self.electra( 2025-10-10T00:55:22.7127577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7128006Z hidden_states = self.encoder( 2025-10-10T00:55:22.7128416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7128838Z layer_outputs = layer_module( 2025-10-10T00:55:22.7129224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7129627Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7130033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7130438Z return func(*args, **kwargs) 2025-10-10T00:55:22.7130850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7131289Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7131977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7132381Z return func(*args, **kwargs) 2025-10-10T00:55:22.7132798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7133358Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7133859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7134304Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7134459Z 2025-10-10T00:55:22.7134577Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7135013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7135392Z res = mod(**inputs) 2025-10-10T00:55:22.7135809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7136234Z outputs = self.electra( 2025-10-10T00:55:22.7136627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7137028Z hidden_states = self.encoder( 2025-10-10T00:55:22.7137437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7137844Z layer_outputs = layer_module( 2025-10-10T00:55:22.7138249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7138620Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7139008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7139393Z return func(*args, **kwargs) 2025-10-10T00:55:22.7139785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7140204Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7140622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7141129Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7141603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7142130Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7142620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7143066Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7143227Z 2025-10-10T00:55:22.7143343Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7144028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7144377Z res = mod(**inputs) 2025-10-10T00:55:22.7144798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7145252Z outputs = self.electra( 2025-10-10T00:55:22.7145667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7146096Z hidden_states = self.encoder( 2025-10-10T00:55:22.7146625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7147089Z layer_outputs = layer_module( 2025-10-10T00:55:22.7147477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7147906Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7148300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7148678Z return func(*args, **kwargs) 2025-10-10T00:55:22.7149081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7149506Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7149918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7150319Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7150777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7151256Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7151698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7152135Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7152502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7152842Z return self.act(input) 2025-10-10T00:55:22.7152962Z 2025-10-10T00:55:22.7153069Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7153431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7153789Z res = mod(**inputs) 2025-10-10T00:55:22.7154156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7154553Z outputs = self.electra( 2025-10-10T00:55:22.7154930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7155328Z hidden_states = self.encoder( 2025-10-10T00:55:22.7155761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7156142Z layer_outputs = layer_module( 2025-10-10T00:55:22.7156552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7156903Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7157278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7157628Z return func(*args, **kwargs) 2025-10-10T00:55:22.7157990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7158380Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7158766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7159148Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7159557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7160037Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7160487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7160896Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7161033Z 2025-10-10T00:55:22.7161148Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7161513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7161853Z res = mod(**inputs) 2025-10-10T00:55:22.7162213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7162597Z outputs = self.electra( 2025-10-10T00:55:22.7162950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7163334Z hidden_states = self.encoder( 2025-10-10T00:55:22.7163709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7164108Z layer_outputs = layer_module( 2025-10-10T00:55:22.7164480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7164830Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7165199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7165569Z return func(*args, **kwargs) 2025-10-10T00:55:22.7165945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7166337Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7166705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7167068Z return func(*args, **kwargs) 2025-10-10T00:55:22.7167438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7167842Z self_outputs = self.self( 2025-10-10T00:55:22.7168201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7168586Z return func(*args, **kwargs) 2025-10-10T00:55:22.7168975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7169395Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7169563Z 2025-10-10T00:55:22.7169681Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7170081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7170429Z res = mod(**inputs) 2025-10-10T00:55:22.7170813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7171223Z outputs = self.electra( 2025-10-10T00:55:22.7171585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7171972Z hidden_states = self.encoder( 2025-10-10T00:55:22.7172359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7172821Z layer_outputs = layer_module( 2025-10-10T00:55:22.7173175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7173534Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7173926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7174302Z return func(*args, **kwargs) 2025-10-10T00:55:22.7174686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7175084Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7175450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7175816Z return func(*args, **kwargs) 2025-10-10T00:55:22.7176190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7176579Z self_outputs = self.self( 2025-10-10T00:55:22.7176934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7177311Z return func(*args, **kwargs) 2025-10-10T00:55:22.7177691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7178091Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7178227Z 2025-10-10T00:55:22.7178347Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7178712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7179047Z res = mod(**inputs) 2025-10-10T00:55:22.7179432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7179836Z outputs = self.electra( 2025-10-10T00:55:22.7180228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7180617Z hidden_states = self.encoder( 2025-10-10T00:55:22.7181005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7181404Z layer_outputs = layer_module( 2025-10-10T00:55:22.7181792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7182160Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7182556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7182935Z return func(*args, **kwargs) 2025-10-10T00:55:22.7183324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7183738Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7184126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7184582Z return func(*args, **kwargs) 2025-10-10T00:55:22.7184994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7185423Z self_outputs = self.self( 2025-10-10T00:55:22.7185813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7186277Z return func(*args, **kwargs) 2025-10-10T00:55:22.7186701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7187138Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7187286Z 2025-10-10T00:55:22.7187390Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7187606Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7187855Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7188235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7188573Z res = mod(**inputs) 2025-10-10T00:55:22.7188959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7189362Z outputs = self.electra( 2025-10-10T00:55:22.7189747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7190152Z hidden_states = self.encoder( 2025-10-10T00:55:22.7190546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7190940Z layer_outputs = layer_module( 2025-10-10T00:55:22.7191296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7191672Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7192067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7192455Z return func(*args, **kwargs) 2025-10-10T00:55:22.7192849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7193267Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7193662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7194052Z return func(*args, **kwargs) 2025-10-10T00:55:22.7194439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7194902Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7195367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7195791Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7195938Z 2025-10-10T00:55:22.7196054Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7196443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7196778Z res = mod(**inputs) 2025-10-10T00:55:22.7197162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7197566Z outputs = self.electra( 2025-10-10T00:55:22.7197954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7198345Z hidden_states = self.encoder( 2025-10-10T00:55:22.7198734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7199186Z layer_outputs = layer_module( 2025-10-10T00:55:22.7199544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7199918Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7200298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7200711Z return func(*args, **kwargs) 2025-10-10T00:55:22.7201123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7201579Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7202006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7202431Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7202894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7203399Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7203867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7204329Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7204488Z 2025-10-10T00:55:22.7204602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7204990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7205349Z res = mod(**inputs) 2025-10-10T00:55:22.7205751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7206184Z outputs = self.electra( 2025-10-10T00:55:22.7206586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7207024Z hidden_states = self.encoder( 2025-10-10T00:55:22.7207448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7207882Z layer_outputs = layer_module( 2025-10-10T00:55:22.7208255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7208656Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7209071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7209473Z return func(*args, **kwargs) 2025-10-10T00:55:22.7209884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7210319Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7210756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7211180Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7211662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7212170Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7212646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7213112Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7213529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7213903Z return self.act(input) 2025-10-10T00:55:22.7214027Z 2025-10-10T00:55:22.7214201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7214632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7214995Z res = mod(**inputs) 2025-10-10T00:55:22.7215400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7215821Z outputs = self.electra( 2025-10-10T00:55:22.7216226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7216662Z hidden_states = self.encoder( 2025-10-10T00:55:22.7217078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7217500Z layer_outputs = layer_module( 2025-10-10T00:55:22.7217868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7218272Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7218683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7219089Z return func(*args, **kwargs) 2025-10-10T00:55:22.7219499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7219934Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7220371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7220807Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7221280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7221813Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7222318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7222770Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7222928Z 2025-10-10T00:55:22.7223055Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7223455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7223805Z res = mod(**inputs) 2025-10-10T00:55:22.7224216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7224656Z outputs = self.electra( 2025-10-10T00:55:22.7225070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7225503Z hidden_states = self.encoder( 2025-10-10T00:55:22.7225925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7226447Z layer_outputs = layer_module( 2025-10-10T00:55:22.7226841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7227279Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7227699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7228119Z return func(*args, **kwargs) 2025-10-10T00:55:22.7228551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7229006Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7229434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7229903Z return func(*args, **kwargs) 2025-10-10T00:55:22.7230331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7230772Z self_outputs = self.self( 2025-10-10T00:55:22.7231176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7231765Z return func(*args, **kwargs) 2025-10-10T00:55:22.7232199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7232657Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7232811Z 2025-10-10T00:55:22.7232943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7233347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7233701Z res = mod(**inputs) 2025-10-10T00:55:22.7234127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7234560Z outputs = self.electra( 2025-10-10T00:55:22.7234948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7235347Z hidden_states = self.encoder( 2025-10-10T00:55:22.7235735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7236134Z layer_outputs = layer_module( 2025-10-10T00:55:22.7236491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7236863Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7237253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7237636Z return func(*args, **kwargs) 2025-10-10T00:55:22.7238027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7238440Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7238832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7239204Z return func(*args, **kwargs) 2025-10-10T00:55:22.7239591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7239991Z self_outputs = self.self( 2025-10-10T00:55:22.7240361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7240737Z return func(*args, **kwargs) 2025-10-10T00:55:22.7241123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7241536Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7241681Z 2025-10-10T00:55:22.7241790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7242219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7242544Z res = mod(**inputs) 2025-10-10T00:55:22.7242932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7243341Z outputs = self.electra( 2025-10-10T00:55:22.7243722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7244126Z hidden_states = self.encoder( 2025-10-10T00:55:22.7244516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7245012Z layer_outputs = layer_module( 2025-10-10T00:55:22.7245369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7245743Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7246132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7246510Z return func(*args, **kwargs) 2025-10-10T00:55:22.7246888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7247289Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7247669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7248034Z return func(*args, **kwargs) 2025-10-10T00:55:22.7248419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7248810Z self_outputs = self.self( 2025-10-10T00:55:22.7249170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7249542Z return func(*args, **kwargs) 2025-10-10T00:55:22.7249914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7250318Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7250461Z 2025-10-10T00:55:22.7250546Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7250768Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7251000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7251363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7251686Z res = mod(**inputs) 2025-10-10T00:55:22.7252063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7252457Z outputs = self.electra( 2025-10-10T00:55:22.7252827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7253217Z hidden_states = self.encoder( 2025-10-10T00:55:22.7253599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7253986Z layer_outputs = layer_module( 2025-10-10T00:55:22.7254324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7254686Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7255062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7255445Z return func(*args, **kwargs) 2025-10-10T00:55:22.7255832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7256265Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7256641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7257006Z return func(*args, **kwargs) 2025-10-10T00:55:22.7257384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7257830Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7258265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7258668Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7258865Z 2025-10-10T00:55:22.7258973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7259335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7259662Z res = mod(**inputs) 2025-10-10T00:55:22.7260023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7260413Z outputs = self.electra( 2025-10-10T00:55:22.7260783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7261172Z hidden_states = self.encoder( 2025-10-10T00:55:22.7261558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7261958Z layer_outputs = layer_module( 2025-10-10T00:55:22.7262316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7262687Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7263072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7263445Z return func(*args, **kwargs) 2025-10-10T00:55:22.7263833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7264244Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7264650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7265050Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7265473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7265961Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7266490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7266922Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7267074Z 2025-10-10T00:55:22.7267190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7267591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7267944Z res = mod(**inputs) 2025-10-10T00:55:22.7268359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7268784Z outputs = self.electra( 2025-10-10T00:55:22.7269167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7269570Z hidden_states = self.encoder( 2025-10-10T00:55:22.7269973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7270380Z layer_outputs = layer_module( 2025-10-10T00:55:22.7270782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7271159Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7271552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7271941Z return func(*args, **kwargs) 2025-10-10T00:55:22.7272333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7272745Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7273207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7273626Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7274070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7274567Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7275016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7275465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7275868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7276228Z return self.act(input) 2025-10-10T00:55:22.7276345Z 2025-10-10T00:55:22.7276459Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7276833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7277162Z res = mod(**inputs) 2025-10-10T00:55:22.7277537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7277932Z outputs = self.electra( 2025-10-10T00:55:22.7278296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7278681Z hidden_states = self.encoder( 2025-10-10T00:55:22.7279063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7279447Z layer_outputs = layer_module( 2025-10-10T00:55:22.7279791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7280142Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7280521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7280890Z return func(*args, **kwargs) 2025-10-10T00:55:22.7281266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7281670Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7282058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7282453Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7282887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7283378Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7283831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7284244Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7284391Z 2025-10-10T00:55:22.7284498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7284917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7285249Z res = mod(**inputs) 2025-10-10T00:55:22.7285609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7285996Z outputs = self.electra( 2025-10-10T00:55:22.7286374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7286757Z hidden_states = self.encoder( 2025-10-10T00:55:22.7287135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7287568Z layer_outputs = layer_module( 2025-10-10T00:55:22.7287908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7288261Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7288632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7288984Z return func(*args, **kwargs) 2025-10-10T00:55:22.7289349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7289745Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7290131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7290503Z return func(*args, **kwargs) 2025-10-10T00:55:22.7290885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7291268Z self_outputs = self.self( 2025-10-10T00:55:22.7291619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7291981Z return func(*args, **kwargs) 2025-10-10T00:55:22.7292350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7292732Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7292874Z 2025-10-10T00:55:22.7292976Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7293324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7293637Z res = mod(**inputs) 2025-10-10T00:55:22.7293990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7294375Z outputs = self.electra( 2025-10-10T00:55:22.7294738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7295119Z hidden_states = self.encoder( 2025-10-10T00:55:22.7295488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7295857Z layer_outputs = layer_module( 2025-10-10T00:55:22.7296193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7296546Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7296914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7297273Z return func(*args, **kwargs) 2025-10-10T00:55:22.7297636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7298028Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7298397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7298780Z return func(*args, **kwargs) 2025-10-10T00:55:22.7299141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7299522Z self_outputs = self.self( 2025-10-10T00:55:22.7299873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7300233Z return func(*args, **kwargs) 2025-10-10T00:55:22.7300598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7300988Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7301182Z 2025-10-10T00:55:22.7301298Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7301646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7301965Z res = mod(**inputs) 2025-10-10T00:55:22.7302368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7302760Z outputs = self.electra( 2025-10-10T00:55:22.7303138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7303528Z hidden_states = self.encoder( 2025-10-10T00:55:22.7303912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7304296Z layer_outputs = layer_module( 2025-10-10T00:55:22.7304652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7305029Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7305417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7305797Z return func(*args, **kwargs) 2025-10-10T00:55:22.7306248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7306677Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7307070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7307456Z return func(*args, **kwargs) 2025-10-10T00:55:22.7307862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7308287Z self_outputs = self.self( 2025-10-10T00:55:22.7308682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7309079Z return func(*args, **kwargs) 2025-10-10T00:55:22.7309490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7309919Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7310070Z 2025-10-10T00:55:22.7310160Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7310394Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7310654Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7311035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7311382Z res = mod(**inputs) 2025-10-10T00:55:22.7311765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7312168Z outputs = self.electra( 2025-10-10T00:55:22.7312561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7313002Z hidden_states = self.encoder( 2025-10-10T00:55:22.7313422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7313842Z layer_outputs = layer_module( 2025-10-10T00:55:22.7314229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7314603Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7314983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7315364Z return func(*args, **kwargs) 2025-10-10T00:55:22.7315789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7316237Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7316626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7317003Z return func(*args, **kwargs) 2025-10-10T00:55:22.7317394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7317860Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7318346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7318784Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7318944Z 2025-10-10T00:55:22.7319060Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7319456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7319808Z res = mod(**inputs) 2025-10-10T00:55:22.7320214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7320635Z outputs = self.electra( 2025-10-10T00:55:22.7321044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7321480Z hidden_states = self.encoder( 2025-10-10T00:55:22.7321874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7322269Z layer_outputs = layer_module( 2025-10-10T00:55:22.7322624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7323004Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7323424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7323830Z return func(*args, **kwargs) 2025-10-10T00:55:22.7324238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7324682Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7325114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7325544Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7326004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7326507Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7326986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7327427Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7327579Z 2025-10-10T00:55:22.7327725Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7328119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7328464Z res = mod(**inputs) 2025-10-10T00:55:22.7328866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7329290Z outputs = self.electra( 2025-10-10T00:55:22.7329697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7330112Z hidden_states = self.encoder( 2025-10-10T00:55:22.7330575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7330989Z layer_outputs = layer_module( 2025-10-10T00:55:22.7331345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7331916Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7332331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7332715Z return func(*args, **kwargs) 2025-10-10T00:55:22.7333106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7333529Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7333960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7334387Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7334855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7335367Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7335832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7336264Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7336663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7337034Z return self.act(input) 2025-10-10T00:55:22.7337156Z 2025-10-10T00:55:22.7337278Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7337669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7338013Z res = mod(**inputs) 2025-10-10T00:55:22.7338421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7338850Z outputs = self.electra( 2025-10-10T00:55:22.7339255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7339684Z hidden_states = self.encoder( 2025-10-10T00:55:22.7340095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7340517Z layer_outputs = layer_module( 2025-10-10T00:55:22.7340892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7341285Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7341691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7342103Z return func(*args, **kwargs) 2025-10-10T00:55:22.7342518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7343021Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7343459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7343878Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7344342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7344871Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7345366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7345804Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7345984Z 2025-10-10T00:55:22.7346209Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7346625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7346979Z res = mod(**inputs) 2025-10-10T00:55:22.7347384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7347808Z outputs = self.electra( 2025-10-10T00:55:22.7348218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7348685Z hidden_states = self.encoder( 2025-10-10T00:55:22.7349104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7349530Z layer_outputs = layer_module( 2025-10-10T00:55:22.7349904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7350304Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7350713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7351120Z return func(*args, **kwargs) 2025-10-10T00:55:22.7351534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7351972Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7352385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7352783Z return func(*args, **kwargs) 2025-10-10T00:55:22.7353196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7353607Z self_outputs = self.self( 2025-10-10T00:55:22.7354005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7354400Z return func(*args, **kwargs) 2025-10-10T00:55:22.7354814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7355221Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7355362Z 2025-10-10T00:55:22.7355470Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7355839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7356163Z res = mod(**inputs) 2025-10-10T00:55:22.7356542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7356944Z outputs = self.electra( 2025-10-10T00:55:22.7357322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7357726Z hidden_states = self.encoder( 2025-10-10T00:55:22.7358119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7358541Z layer_outputs = layer_module( 2025-10-10T00:55:22.7358892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7359265Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7359656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7360032Z return func(*args, **kwargs) 2025-10-10T00:55:22.7360418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7360818Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7361265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7361642Z return func(*args, **kwargs) 2025-10-10T00:55:22.7362032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7362421Z self_outputs = self.self( 2025-10-10T00:55:22.7362793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7363170Z return func(*args, **kwargs) 2025-10-10T00:55:22.7363556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7363961Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7364099Z 2025-10-10T00:55:22.7364204Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7364582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7364914Z res = mod(**inputs) 2025-10-10T00:55:22.7365294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7365699Z outputs = self.electra( 2025-10-10T00:55:22.7366076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7366475Z hidden_states = self.encoder( 2025-10-10T00:55:22.7366868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7367262Z layer_outputs = layer_module( 2025-10-10T00:55:22.7367611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7367986Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7368377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7368753Z return func(*args, **kwargs) 2025-10-10T00:55:22.7369143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7369548Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7369941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7370319Z return func(*args, **kwargs) 2025-10-10T00:55:22.7370704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7371100Z self_outputs = self.self( 2025-10-10T00:55:22.7371468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7371836Z return func(*args, **kwargs) 2025-10-10T00:55:22.7372212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7372630Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7372765Z 2025-10-10T00:55:22.7372846Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7373065Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7373306Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7373664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7373988Z res = mod(**inputs) 2025-10-10T00:55:22.7374355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7374750Z outputs = self.electra( 2025-10-10T00:55:22.7375187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7375582Z hidden_states = self.encoder( 2025-10-10T00:55:22.7375963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7376353Z layer_outputs = layer_module( 2025-10-10T00:55:22.7376701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7377067Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7377444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7377810Z return func(*args, **kwargs) 2025-10-10T00:55:22.7378186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7378591Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7378991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7379373Z return func(*args, **kwargs) 2025-10-10T00:55:22.7379766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7380249Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7380718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7381130Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7381275Z 2025-10-10T00:55:22.7381384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7381753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7382091Z res = mod(**inputs) 2025-10-10T00:55:22.7382495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7382923Z outputs = self.electra( 2025-10-10T00:55:22.7383319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7383741Z hidden_states = self.encoder( 2025-10-10T00:55:22.7384158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7384580Z layer_outputs = layer_module( 2025-10-10T00:55:22.7384952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7385336Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7385753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7386223Z return func(*args, **kwargs) 2025-10-10T00:55:22.7386651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7387126Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7387538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7387945Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7388378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7388857Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7389298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7389766Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7389919Z 2025-10-10T00:55:22.7390028Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7390396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7390728Z res = mod(**inputs) 2025-10-10T00:55:22.7391101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7391503Z outputs = self.electra( 2025-10-10T00:55:22.7391881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7392283Z hidden_states = self.encoder( 2025-10-10T00:55:22.7392667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7393062Z layer_outputs = layer_module( 2025-10-10T00:55:22.7393421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7393795Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7394187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7394559Z return func(*args, **kwargs) 2025-10-10T00:55:22.7394948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7395359Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7395768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7396165Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7396590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7397096Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7397572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7398010Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7398423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7398845Z return self.act(input) 2025-10-10T00:55:22.7398968Z 2025-10-10T00:55:22.7399076Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7399445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7399777Z res = mod(**inputs) 2025-10-10T00:55:22.7400148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7400553Z outputs = self.electra( 2025-10-10T00:55:22.7400934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7401357Z hidden_states = self.encoder( 2025-10-10T00:55:22.7401805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7402202Z layer_outputs = layer_module( 2025-10-10T00:55:22.7402557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7402928Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7403318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7403696Z return func(*args, **kwargs) 2025-10-10T00:55:22.7404108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7404527Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7404930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7405320Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7405739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7406220Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7406675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7407086Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7407228Z 2025-10-10T00:55:22.7407342Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7407707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7408040Z res = mod(**inputs) 2025-10-10T00:55:22.7408418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7408822Z outputs = self.electra( 2025-10-10T00:55:22.7409205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7409599Z hidden_states = self.encoder( 2025-10-10T00:55:22.7409992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7410391Z layer_outputs = layer_module( 2025-10-10T00:55:22.7410746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7411116Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7411518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7411925Z return func(*args, **kwargs) 2025-10-10T00:55:22.7412331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7412745Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7413133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7413516Z return func(*args, **kwargs) 2025-10-10T00:55:22.7413908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7414309Z self_outputs = self.self( 2025-10-10T00:55:22.7414671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7415053Z return func(*args, **kwargs) 2025-10-10T00:55:22.7415439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7415870Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7416010Z 2025-10-10T00:55:22.7416125Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7416486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7416820Z res = mod(**inputs) 2025-10-10T00:55:22.7417199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7417602Z outputs = self.electra( 2025-10-10T00:55:22.7417987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7418399Z hidden_states = self.encoder( 2025-10-10T00:55:22.7418834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7419236Z layer_outputs = layer_module( 2025-10-10T00:55:22.7419603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7419989Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7420400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7420795Z return func(*args, **kwargs) 2025-10-10T00:55:22.7421207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7421642Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7422049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7422451Z return func(*args, **kwargs) 2025-10-10T00:55:22.7422860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7423287Z self_outputs = self.self( 2025-10-10T00:55:22.7423676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7424066Z return func(*args, **kwargs) 2025-10-10T00:55:22.7424474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7424903Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7425044Z 2025-10-10T00:55:22.7425164Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7425549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7425902Z res = mod(**inputs) 2025-10-10T00:55:22.7426376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7426817Z outputs = self.electra( 2025-10-10T00:55:22.7427221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7427637Z hidden_states = self.encoder( 2025-10-10T00:55:22.7428053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7428476Z layer_outputs = layer_module( 2025-10-10T00:55:22.7428851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7429251Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7429662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7430067Z return func(*args, **kwargs) 2025-10-10T00:55:22.7430480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7430939Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7431325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7431922Z return func(*args, **kwargs) 2025-10-10T00:55:22.7432346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7432776Z self_outputs = self.self( 2025-10-10T00:55:22.7433168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7433576Z return func(*args, **kwargs) 2025-10-10T00:55:22.7434129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7434542Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7434682Z 2025-10-10T00:55:22.7434773Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7434989Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7435237Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7435606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7435939Z res = mod(**inputs) 2025-10-10T00:55:22.7436325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7436722Z outputs = self.electra( 2025-10-10T00:55:22.7437106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7437515Z hidden_states = self.encoder( 2025-10-10T00:55:22.7437911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7438300Z layer_outputs = layer_module( 2025-10-10T00:55:22.7438662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7439023Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7439404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7439774Z return func(*args, **kwargs) 2025-10-10T00:55:22.7440140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7440540Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7440920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7441288Z return func(*args, **kwargs) 2025-10-10T00:55:22.7441672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7442127Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7442568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7442968Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7443104Z 2025-10-10T00:55:22.7443217Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7443575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7443889Z res = mod(**inputs) 2025-10-10T00:55:22.7444263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7444659Z outputs = self.electra( 2025-10-10T00:55:22.7445029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7445452Z hidden_states = self.encoder( 2025-10-10T00:55:22.7445838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7446230Z layer_outputs = layer_module( 2025-10-10T00:55:22.7446579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7446946Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7447319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7447690Z return func(*args, **kwargs) 2025-10-10T00:55:22.7448126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7448536Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7448932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7449327Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7449754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7450222Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7450657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7451064Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7451212Z 2025-10-10T00:55:22.7451321Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7451677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7451996Z res = mod(**inputs) 2025-10-10T00:55:22.7452367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7452753Z outputs = self.electra( 2025-10-10T00:55:22.7453126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7453511Z hidden_states = self.encoder( 2025-10-10T00:55:22.7453894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7454284Z layer_outputs = layer_module( 2025-10-10T00:55:22.7454613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7454970Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7455337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7455696Z return func(*args, **kwargs) 2025-10-10T00:55:22.7456059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7456448Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7456831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7457212Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7457620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7458067Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7458507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7458942Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7459353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7459705Z return self.act(input) 2025-10-10T00:55:22.7459821Z 2025-10-10T00:55:22.7459928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7460304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7460661Z res = mod(**inputs) 2025-10-10T00:55:22.7461064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7461481Z outputs = self.electra( 2025-10-10T00:55:22.7461931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7462356Z hidden_states = self.encoder( 2025-10-10T00:55:22.7462773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7463207Z layer_outputs = layer_module( 2025-10-10T00:55:22.7463573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7463971Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7464389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7464799Z return func(*args, **kwargs) 2025-10-10T00:55:22.7465245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7465651Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7466063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7466543Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7467018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7467524Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7467985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7468378Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7468518Z 2025-10-10T00:55:22.7468634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7468998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7469325Z res = mod(**inputs) 2025-10-10T00:55:22.7469713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7470122Z outputs = self.electra( 2025-10-10T00:55:22.7470518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7470913Z hidden_states = self.encoder( 2025-10-10T00:55:22.7471292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7471687Z layer_outputs = layer_module( 2025-10-10T00:55:22.7489327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7489758Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7490200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7490615Z return func(*args, **kwargs) 2025-10-10T00:55:22.7491033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7491561Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7491980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7492382Z return func(*args, **kwargs) 2025-10-10T00:55:22.7492793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7493219Z self_outputs = self.self( 2025-10-10T00:55:22.7493600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7493996Z return func(*args, **kwargs) 2025-10-10T00:55:22.7494467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7494910Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7495059Z 2025-10-10T00:55:22.7495180Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7495566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7495912Z res = mod(**inputs) 2025-10-10T00:55:22.7496310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7496726Z outputs = self.electra( 2025-10-10T00:55:22.7497116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7497530Z hidden_states = self.encoder( 2025-10-10T00:55:22.7497940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7498349Z layer_outputs = layer_module( 2025-10-10T00:55:22.7498705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7499091Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7499498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7499923Z return func(*args, **kwargs) 2025-10-10T00:55:22.7500316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7500726Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7501126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7501511Z return func(*args, **kwargs) 2025-10-10T00:55:22.7501922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7502347Z self_outputs = self.self( 2025-10-10T00:55:22.7502746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7503162Z return func(*args, **kwargs) 2025-10-10T00:55:22.7503579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7504014Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7504163Z 2025-10-10T00:55:22.7504292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7504688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7505045Z res = mod(**inputs) 2025-10-10T00:55:22.7505473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7505910Z outputs = self.electra( 2025-10-10T00:55:22.7506417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7506906Z hidden_states = self.encoder( 2025-10-10T00:55:22.7507434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7507841Z layer_outputs = layer_module( 2025-10-10T00:55:22.7508230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7508634Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7509071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7509499Z return func(*args, **kwargs) 2025-10-10T00:55:22.7509996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7510439Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7510878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7511292Z return func(*args, **kwargs) 2025-10-10T00:55:22.7511717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7512152Z self_outputs = self.self( 2025-10-10T00:55:22.7512550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7512975Z return func(*args, **kwargs) 2025-10-10T00:55:22.7513404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7513862Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7514014Z 2025-10-10T00:55:22.7514118Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7514358Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7514634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7515045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7515430Z res = mod(**inputs) 2025-10-10T00:55:22.7515842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7516286Z outputs = self.electra( 2025-10-10T00:55:22.7516582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7516665Z hidden_states = self.encoder( 2025-10-10T00:55:22.7516957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7517048Z layer_outputs = layer_module( 2025-10-10T00:55:22.7517294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7517391Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7517658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7517738Z return func(*args, **kwargs) 2025-10-10T00:55:22.7518034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7518129Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7518405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7518482Z return func(*args, **kwargs) 2025-10-10T00:55:22.7518781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7518926Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7519238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7519340Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7519345Z 2025-10-10T00:55:22.7519465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7519694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7519769Z res = mod(**inputs) 2025-10-10T00:55:22.7520081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7520169Z outputs = self.electra( 2025-10-10T00:55:22.7520504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7520593Z hidden_states = self.encoder( 2025-10-10T00:55:22.7520882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7520969Z layer_outputs = layer_module( 2025-10-10T00:55:22.7521212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7521300Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7521575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7521653Z return func(*args, **kwargs) 2025-10-10T00:55:22.7521947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7522049Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7522334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7522436Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7522762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7522907Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7523190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7523293Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7523298Z 2025-10-10T00:55:22.7523416Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7523636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7523724Z res = mod(**inputs) 2025-10-10T00:55:22.7524019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7524107Z outputs = self.electra( 2025-10-10T00:55:22.7524393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7524474Z hidden_states = self.encoder( 2025-10-10T00:55:22.7524770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7524849Z layer_outputs = layer_module( 2025-10-10T00:55:22.7525100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7525187Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7525457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7525547Z return func(*args, **kwargs) 2025-10-10T00:55:22.7525833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7525984Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7526265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7526362Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7526685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7526820Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7527120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7527298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7527547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7527628Z return self.act(input) 2025-10-10T00:55:22.7527633Z 2025-10-10T00:55:22.7527749Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7527979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7528053Z res = mod(**inputs) 2025-10-10T00:55:22.7528356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7528434Z outputs = self.electra( 2025-10-10T00:55:22.7528733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7528816Z hidden_states = self.encoder( 2025-10-10T00:55:22.7529111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7529205Z layer_outputs = layer_module( 2025-10-10T00:55:22.7529452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7529548Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7529816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7529895Z return func(*args, **kwargs) 2025-10-10T00:55:22.7530196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7530291Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7530591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7530681Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7531004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7531166Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7531619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7531734Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7531738Z 2025-10-10T00:55:22.7531855Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7532088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7532163Z res = mod(**inputs) 2025-10-10T00:55:22.7532463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7532559Z outputs = self.electra( 2025-10-10T00:55:22.7532844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7532998Z hidden_states = self.encoder( 2025-10-10T00:55:22.7533289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7533372Z layer_outputs = layer_module( 2025-10-10T00:55:22.7533633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7533722Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7534003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7534089Z return func(*args, **kwargs) 2025-10-10T00:55:22.7534455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7534540Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7534782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7534863Z return func(*args, **kwargs) 2025-10-10T00:55:22.7535122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7535204Z self_outputs = self.self( 2025-10-10T00:55:22.7535447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7535519Z return func(*args, **kwargs) 2025-10-10T00:55:22.7535788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7535876Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7535880Z 2025-10-10T00:55:22.7535994Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7536195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7536272Z res = mod(**inputs) 2025-10-10T00:55:22.7536537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7536609Z outputs = self.electra( 2025-10-10T00:55:22.7536873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7536947Z hidden_states = self.encoder( 2025-10-10T00:55:22.7537215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7537290Z layer_outputs = layer_module( 2025-10-10T00:55:22.7537516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7537608Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7537852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7537933Z return func(*args, **kwargs) 2025-10-10T00:55:22.7538193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7538277Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7538525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7538597Z return func(*args, **kwargs) 2025-10-10T00:55:22.7538863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7538943Z self_outputs = self.self( 2025-10-10T00:55:22.7539194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7539285Z return func(*args, **kwargs) 2025-10-10T00:55:22.7539547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7539640Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7539643Z 2025-10-10T00:55:22.7539750Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7539958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7540027Z res = mod(**inputs) 2025-10-10T00:55:22.7540294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7540376Z outputs = self.electra( 2025-10-10T00:55:22.7540685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7540765Z hidden_states = self.encoder( 2025-10-10T00:55:22.7541032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7541106Z layer_outputs = layer_module( 2025-10-10T00:55:22.7541353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7541438Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7541708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7541782Z return func(*args, **kwargs) 2025-10-10T00:55:22.7542072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7542166Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7542425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7542511Z return func(*args, **kwargs) 2025-10-10T00:55:22.7542791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7542877Z self_outputs = self.self( 2025-10-10T00:55:22.7543136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7543211Z return func(*args, **kwargs) 2025-10-10T00:55:22.7543500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7543588Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7543592Z 2025-10-10T00:55:22.7543695Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7543782Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7543895Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7544119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7544189Z res = mod(**inputs) 2025-10-10T00:55:22.7544486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7544562Z outputs = self.electra( 2025-10-10T00:55:22.7544877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7544953Z hidden_states = self.encoder( 2025-10-10T00:55:22.7545222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7545306Z layer_outputs = layer_module( 2025-10-10T00:55:22.7545548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7545642Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7545935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7546011Z return func(*args, **kwargs) 2025-10-10T00:55:22.7546365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7546462Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7546736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7546811Z return func(*args, **kwargs) 2025-10-10T00:55:22.7547137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7547316Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7547601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7547701Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7547705Z 2025-10-10T00:55:22.7547810Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7548019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7548087Z res = mod(**inputs) 2025-10-10T00:55:22.7548357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7548440Z outputs = self.electra( 2025-10-10T00:55:22.7548764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7548852Z hidden_states = self.encoder( 2025-10-10T00:55:22.7549116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7549191Z layer_outputs = layer_module( 2025-10-10T00:55:22.7549424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7549502Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7549758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7549829Z return func(*args, **kwargs) 2025-10-10T00:55:22.7550104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7550189Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7550447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7550545Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7550827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7550956Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7551206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7551289Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7551292Z 2025-10-10T00:55:22.7551400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7551591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7551663Z res = mod(**inputs) 2025-10-10T00:55:22.7551921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7551999Z outputs = self.electra( 2025-10-10T00:55:22.7552249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7552337Z hidden_states = self.encoder( 2025-10-10T00:55:22.7552598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7552670Z layer_outputs = layer_module( 2025-10-10T00:55:22.7552895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7552975Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7553218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7553314Z return func(*args, **kwargs) 2025-10-10T00:55:22.7553605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7553699Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7553952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7554029Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7554335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7554453Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7554709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7554818Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7555036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7555107Z return self.act(input) 2025-10-10T00:55:22.7555111Z 2025-10-10T00:55:22.7555212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7555406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7555475Z res = mod(**inputs) 2025-10-10T00:55:22.7555734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7555801Z outputs = self.electra( 2025-10-10T00:55:22.7556052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7556120Z hidden_states = self.encoder( 2025-10-10T00:55:22.7556374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7556444Z layer_outputs = layer_module( 2025-10-10T00:55:22.7556654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7556739Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7556969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7557044Z return func(*args, **kwargs) 2025-10-10T00:55:22.7557295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7557380Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7557639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7557714Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7558009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7558140Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7558416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7558499Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7558503Z 2025-10-10T00:55:22.7558606Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7558808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7558873Z res = mod(**inputs) 2025-10-10T00:55:22.7559140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7559209Z outputs = self.electra( 2025-10-10T00:55:22.7559526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7559608Z hidden_states = self.encoder( 2025-10-10T00:55:22.7559867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7559944Z layer_outputs = layer_module( 2025-10-10T00:55:22.7560155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7560240Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7560478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7560547Z return func(*args, **kwargs) 2025-10-10T00:55:22.7560808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7560905Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7561140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7561208Z return func(*args, **kwargs) 2025-10-10T00:55:22.7561454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7561531Z self_outputs = self.self( 2025-10-10T00:55:22.7561759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7561833Z return func(*args, **kwargs) 2025-10-10T00:55:22.7562080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7562159Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7562169Z 2025-10-10T00:55:22.7562272Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7562463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7562533Z res = mod(**inputs) 2025-10-10T00:55:22.7562786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7562858Z outputs = self.electra( 2025-10-10T00:55:22.7563104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7563172Z hidden_states = self.encoder( 2025-10-10T00:55:22.7563426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7563494Z layer_outputs = layer_module( 2025-10-10T00:55:22.7563712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7563792Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7564034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7564132Z return func(*args, **kwargs) 2025-10-10T00:55:22.7564392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7564491Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7564727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7564795Z return func(*args, **kwargs) 2025-10-10T00:55:22.7565055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7565126Z self_outputs = self.self( 2025-10-10T00:55:22.7565401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7565486Z return func(*args, **kwargs) 2025-10-10T00:55:22.7565750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7565831Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7565835Z 2025-10-10T00:55:22.7565938Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7566143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7566209Z res = mod(**inputs) 2025-10-10T00:55:22.7566476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7566545Z outputs = self.electra( 2025-10-10T00:55:22.7566805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7566885Z hidden_states = self.encoder( 2025-10-10T00:55:22.7567142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7567222Z layer_outputs = layer_module( 2025-10-10T00:55:22.7567441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7567527Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7567767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7567838Z return func(*args, **kwargs) 2025-10-10T00:55:22.7568099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7568180Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7568429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7568497Z return func(*args, **kwargs) 2025-10-10T00:55:22.7568758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7568841Z self_outputs = self.self( 2025-10-10T00:55:22.7569089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7569166Z return func(*args, **kwargs) 2025-10-10T00:55:22.7569428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7569511Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7569521Z 2025-10-10T00:55:22.7569604Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7569687Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7569804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7570005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7570093Z res = mod(**inputs) 2025-10-10T00:55:22.7570362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7570433Z outputs = self.electra( 2025-10-10T00:55:22.7570700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7570771Z hidden_states = self.encoder( 2025-10-10T00:55:22.7571038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7571110Z layer_outputs = layer_module( 2025-10-10T00:55:22.7571368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7571475Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7571716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7571796Z return func(*args, **kwargs) 2025-10-10T00:55:22.7572055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7572138Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7572388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7572458Z return func(*args, **kwargs) 2025-10-10T00:55:22.7572727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7572858Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7573128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7573215Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7573221Z 2025-10-10T00:55:22.7573327Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7573535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7573602Z res = mod(**inputs) 2025-10-10T00:55:22.7573874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7573944Z outputs = self.electra( 2025-10-10T00:55:22.7574205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7574285Z hidden_states = self.encoder( 2025-10-10T00:55:22.7574547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7574629Z layer_outputs = layer_module( 2025-10-10T00:55:22.7574852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7574934Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7575191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7575260Z return func(*args, **kwargs) 2025-10-10T00:55:22.7575520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7575603Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7575859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7575940Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7576226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7576376Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7576631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7576720Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7576724Z 2025-10-10T00:55:22.7576826Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7577020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7577091Z res = mod(**inputs) 2025-10-10T00:55:22.7577349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7578837Z outputs = self.electra( 2025-10-10T00:55:22.7579107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7579186Z hidden_states = self.encoder( 2025-10-10T00:55:22.7579437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7579508Z layer_outputs = layer_module( 2025-10-10T00:55:22.7579731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7579812Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7580062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7580134Z return func(*args, **kwargs) 2025-10-10T00:55:22.7580397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7580495Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7580753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7580842Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7581145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7581280Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7581557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7581680Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7581917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7581998Z return self.act(input) 2025-10-10T00:55:22.7582003Z 2025-10-10T00:55:22.7582121Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7582331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7582404Z res = mod(**inputs) 2025-10-10T00:55:22.7582693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7582766Z outputs = self.electra( 2025-10-10T00:55:22.7583047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7583124Z hidden_states = self.encoder( 2025-10-10T00:55:22.7583399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7583483Z layer_outputs = layer_module( 2025-10-10T00:55:22.7583724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7583820Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7584105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7584187Z return func(*args, **kwargs) 2025-10-10T00:55:22.7584463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7584554Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7584838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7584922Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7585241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7585457Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7585733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7585831Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7585835Z 2025-10-10T00:55:22.7585947Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7586239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7586321Z res = mod(**inputs) 2025-10-10T00:55:22.7586653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7586730Z outputs = self.electra( 2025-10-10T00:55:22.7587011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7587103Z hidden_states = self.encoder( 2025-10-10T00:55:22.7587380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7587467Z layer_outputs = layer_module( 2025-10-10T00:55:22.7587705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7587792Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7588072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7588150Z return func(*args, **kwargs) 2025-10-10T00:55:22.7588489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7588581Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7588862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7588939Z return func(*args, **kwargs) 2025-10-10T00:55:22.7589218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7589308Z self_outputs = self.self( 2025-10-10T00:55:22.7589579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7589662Z return func(*args, **kwargs) 2025-10-10T00:55:22.7589940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7590030Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7590034Z 2025-10-10T00:55:22.7590155Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7590372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7590451Z res = mod(**inputs) 2025-10-10T00:55:22.7590744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7590842Z outputs = self.electra( 2025-10-10T00:55:22.7591126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7591203Z hidden_states = self.encoder( 2025-10-10T00:55:22.7591488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7591566Z layer_outputs = layer_module( 2025-10-10T00:55:22.7591811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7591897Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7592204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7592305Z return func(*args, **kwargs) 2025-10-10T00:55:22.7592586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7592683Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7592951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7593024Z return func(*args, **kwargs) 2025-10-10T00:55:22.7593312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7593389Z self_outputs = self.self( 2025-10-10T00:55:22.7593652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7593726Z return func(*args, **kwargs) 2025-10-10T00:55:22.7594011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7594103Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7594108Z 2025-10-10T00:55:22.7594220Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7594443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7594514Z res = mod(**inputs) 2025-10-10T00:55:22.7594804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7594878Z outputs = self.electra( 2025-10-10T00:55:22.7595155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7595236Z hidden_states = self.encoder( 2025-10-10T00:55:22.7595521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7595605Z layer_outputs = layer_module( 2025-10-10T00:55:22.7595844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7595929Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7596210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7596283Z return func(*args, **kwargs) 2025-10-10T00:55:22.7596565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7596653Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7596915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7596992Z return func(*args, **kwargs) 2025-10-10T00:55:22.7597268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7597371Z self_outputs = self.self( 2025-10-10T00:55:22.7597628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7597708Z return func(*args, **kwargs) 2025-10-10T00:55:22.7597986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7598072Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7598076Z 2025-10-10T00:55:22.7598171Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7598257Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7598375Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7598617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7598700Z res = mod(**inputs) 2025-10-10T00:55:22.7598977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7599049Z outputs = self.electra( 2025-10-10T00:55:22.7599318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7599391Z hidden_states = self.encoder( 2025-10-10T00:55:22.7599653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7599735Z layer_outputs = layer_module( 2025-10-10T00:55:22.7599960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7600046Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7600293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7600372Z return func(*args, **kwargs) 2025-10-10T00:55:22.7600635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7600720Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7600972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7601042Z return func(*args, **kwargs) 2025-10-10T00:55:22.7601310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7601443Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7601705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7601807Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7601810Z 2025-10-10T00:55:22.7601923Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7602123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7602196Z res = mod(**inputs) 2025-10-10T00:55:22.7602463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7602532Z outputs = self.electra( 2025-10-10T00:55:22.7602805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7602877Z hidden_states = self.encoder( 2025-10-10T00:55:22.7603142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7603234Z layer_outputs = layer_module( 2025-10-10T00:55:22.7603458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7603564Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7603805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7603875Z return func(*args, **kwargs) 2025-10-10T00:55:22.7604145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7604232Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7604498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7604579Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7604918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7605061Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7605329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7605424Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7605427Z 2025-10-10T00:55:22.7605532Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7605741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7605830Z res = mod(**inputs) 2025-10-10T00:55:22.7606101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7606180Z outputs = self.electra( 2025-10-10T00:55:22.7606449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7606529Z hidden_states = self.encoder( 2025-10-10T00:55:22.7606795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7606877Z layer_outputs = layer_module( 2025-10-10T00:55:22.7607104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7607185Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7607439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7607511Z return func(*args, **kwargs) 2025-10-10T00:55:22.7607784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7607869Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7608137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7608223Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7608525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7608654Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7608921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7609043Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7609263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7609334Z return self.act(input) 2025-10-10T00:55:22.7609338Z 2025-10-10T00:55:22.7609449Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7609664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7609742Z res = mod(**inputs) 2025-10-10T00:55:22.7610053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7610127Z outputs = self.electra( 2025-10-10T00:55:22.7610406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7610482Z hidden_states = self.encoder( 2025-10-10T00:55:22.7610762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7610839Z layer_outputs = layer_module( 2025-10-10T00:55:22.7611072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7611213Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7611470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7611553Z return func(*args, **kwargs) 2025-10-10T00:55:22.7611827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7611923Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7612193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7612275Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7612592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7612738Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7613038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7613130Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7613135Z 2025-10-10T00:55:22.7613248Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7613467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7613537Z res = mod(**inputs) 2025-10-10T00:55:22.7613837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7613908Z outputs = self.electra( 2025-10-10T00:55:22.7614181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7614256Z hidden_states = self.encoder( 2025-10-10T00:55:22.7614539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7614626Z layer_outputs = layer_module( 2025-10-10T00:55:22.7614864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7614958Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7615221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7615297Z return func(*args, **kwargs) 2025-10-10T00:55:22.7615585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7615674Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7615941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7616016Z return func(*args, **kwargs) 2025-10-10T00:55:22.7616306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7616387Z self_outputs = self.self( 2025-10-10T00:55:22.7616646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7616724Z return func(*args, **kwargs) 2025-10-10T00:55:22.7616988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7617080Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7617084Z 2025-10-10T00:55:22.7617189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7617387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7617462Z res = mod(**inputs) 2025-10-10T00:55:22.7617759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7617857Z outputs = self.electra( 2025-10-10T00:55:22.7618117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7618191Z hidden_states = self.encoder( 2025-10-10T00:55:22.7618460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7618534Z layer_outputs = layer_module( 2025-10-10T00:55:22.7618776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7618860Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7619117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7619204Z return func(*args, **kwargs) 2025-10-10T00:55:22.7619483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7619581Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7619840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7619919Z return func(*args, **kwargs) 2025-10-10T00:55:22.7620198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7620274Z self_outputs = self.self( 2025-10-10T00:55:22.7620538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7620611Z return func(*args, **kwargs) 2025-10-10T00:55:22.7620900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7620987Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7620991Z 2025-10-10T00:55:22.7621101Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7621320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7621391Z res = mod(**inputs) 2025-10-10T00:55:22.7621677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7621751Z outputs = self.electra( 2025-10-10T00:55:22.7622035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7622111Z hidden_states = self.encoder( 2025-10-10T00:55:22.7622388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7622481Z layer_outputs = layer_module( 2025-10-10T00:55:22.7622716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7622828Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7623084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7623156Z return func(*args, **kwargs) 2025-10-10T00:55:22.7623441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7623531Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7623795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7623869Z return func(*args, **kwargs) 2025-10-10T00:55:22.7624179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7624281Z self_outputs = self.self( 2025-10-10T00:55:22.7624537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7624618Z return func(*args, **kwargs) 2025-10-10T00:55:22.7624896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7624989Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7624993Z 2025-10-10T00:55:22.7625084Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7625172Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7625294Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7625511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7625590Z res = mod(**inputs) 2025-10-10T00:55:22.7625886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7625962Z outputs = self.electra( 2025-10-10T00:55:22.7626514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7626600Z hidden_states = self.encoder( 2025-10-10T00:55:22.7626893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7626971Z layer_outputs = layer_module( 2025-10-10T00:55:22.7627214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7627310Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7627573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7627664Z return func(*args, **kwargs) 2025-10-10T00:55:22.7627950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7628049Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7628326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7628402Z return func(*args, **kwargs) 2025-10-10T00:55:22.7628698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7628842Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7629136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7629231Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7629237Z 2025-10-10T00:55:22.7629354Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7629580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7629687Z res = mod(**inputs) 2025-10-10T00:55:22.7629987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7630063Z outputs = self.electra( 2025-10-10T00:55:22.7630355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7630433Z hidden_states = self.encoder( 2025-10-10T00:55:22.7630717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7630803Z layer_outputs = layer_module( 2025-10-10T00:55:22.7631080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7631203Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7631626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7631716Z return func(*args, **kwargs) 2025-10-10T00:55:22.7632014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7632111Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7632402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7632490Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7632816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7632964Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7633252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7633355Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7633359Z 2025-10-10T00:55:22.7633471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7633699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7633773Z res = mod(**inputs) 2025-10-10T00:55:22.7634069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7634156Z outputs = self.electra( 2025-10-10T00:55:22.7634445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7634532Z hidden_states = self.encoder( 2025-10-10T00:55:22.7634825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7634906Z layer_outputs = layer_module( 2025-10-10T00:55:22.7635163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7635250Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7635527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7635604Z return func(*args, **kwargs) 2025-10-10T00:55:22.7635875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7635955Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7636197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7636278Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7636550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7636708Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7636948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7637054Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7637262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7637328Z return self.act(input) 2025-10-10T00:55:22.7637332Z 2025-10-10T00:55:22.7637434Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7637619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7637769Z res = mod(**inputs) 2025-10-10T00:55:22.7638093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7638169Z outputs = self.electra( 2025-10-10T00:55:22.7638416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7638494Z hidden_states = self.encoder( 2025-10-10T00:55:22.7638741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7638818Z layer_outputs = layer_module( 2025-10-10T00:55:22.7639029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7639106Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7639347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7639418Z return func(*args, **kwargs) 2025-10-10T00:55:22.7639672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7639755Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7640002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7640086Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7640372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7640514Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7640771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7640867Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7640871Z 2025-10-10T00:55:22.7640974Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7641167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7641239Z res = mod(**inputs) 2025-10-10T00:55:22.7641501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7641575Z outputs = self.electra( 2025-10-10T00:55:22.7641821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7641889Z hidden_states = self.encoder( 2025-10-10T00:55:22.7642141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7642213Z layer_outputs = layer_module( 2025-10-10T00:55:22.7642428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7642504Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7642755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7642828Z return func(*args, **kwargs) 2025-10-10T00:55:22.7643074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7643162Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7643398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7643473Z return func(*args, **kwargs) 2025-10-10T00:55:22.7643760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7643851Z self_outputs = self.self( 2025-10-10T00:55:22.7644102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7644173Z return func(*args, **kwargs) 2025-10-10T00:55:22.7644444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-10-10T00:55:22.7644527Z query_layer = self.query(hidden_states) 2025-10-10T00:55:22.7644531Z 2025-10-10T00:55:22.7644636Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7644843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7644909Z res = mod(**inputs) 2025-10-10T00:55:22.7645183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7645257Z outputs = self.electra( 2025-10-10T00:55:22.7645527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7645601Z hidden_states = self.encoder( 2025-10-10T00:55:22.7645860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7645941Z layer_outputs = layer_module( 2025-10-10T00:55:22.7646163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7646250Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7646494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7646564Z return func(*args, **kwargs) 2025-10-10T00:55:22.7646835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7646922Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7647170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7647242Z return func(*args, **kwargs) 2025-10-10T00:55:22.7647502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7647583Z self_outputs = self.self( 2025-10-10T00:55:22.7647827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7647904Z return func(*args, **kwargs) 2025-10-10T00:55:22.7648163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-10-10T00:55:22.7648255Z key_layer = self.key(current_states) 2025-10-10T00:55:22.7648260Z 2025-10-10T00:55:22.7648375Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7648588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7648686Z res = mod(**inputs) 2025-10-10T00:55:22.7648980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7649057Z outputs = self.electra( 2025-10-10T00:55:22.7649323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7649394Z hidden_states = self.encoder( 2025-10-10T00:55:22.7649671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7649743Z layer_outputs = layer_module( 2025-10-10T00:55:22.7650008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7650112Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7650355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7650434Z return func(*args, **kwargs) 2025-10-10T00:55:22.7650693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7650782Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7651025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7651101Z return func(*args, **kwargs) 2025-10-10T00:55:22.7651359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-10-10T00:55:22.7651430Z self_outputs = self.self( 2025-10-10T00:55:22.7651683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7651753Z return func(*args, **kwargs) 2025-10-10T00:55:22.7652021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-10-10T00:55:22.7652103Z value_layer = self.value(current_states) 2025-10-10T00:55:22.7652106Z 2025-10-10T00:55:22.7652188Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7652286Z cudagraph partition due to non gpu ops 2025-10-10T00:55:22.7652389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7652590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7652654Z res = mod(**inputs) 2025-10-10T00:55:22.7652912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7652990Z outputs = self.electra( 2025-10-10T00:55:22.7653244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7653326Z hidden_states = self.encoder( 2025-10-10T00:55:22.7653585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7653665Z layer_outputs = layer_module( 2025-10-10T00:55:22.7653887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7653965Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7654212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7654283Z return func(*args, **kwargs) 2025-10-10T00:55:22.7654553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-10-10T00:55:22.7654637Z self_attention_outputs = self.attention( 2025-10-10T00:55:22.7654881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7654980Z return func(*args, **kwargs) 2025-10-10T00:55:22.7655246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-10-10T00:55:22.7655383Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:55:22.7655647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-10-10T00:55:22.7655734Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7655744Z 2025-10-10T00:55:22.7655849Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7656106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7656182Z res = mod(**inputs) 2025-10-10T00:55:22.7656489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7656566Z outputs = self.electra( 2025-10-10T00:55:22.7656825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7656897Z hidden_states = self.encoder( 2025-10-10T00:55:22.7657163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7657236Z layer_outputs = layer_module( 2025-10-10T00:55:22.7657465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7657543Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7657794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7657876Z return func(*args, **kwargs) 2025-10-10T00:55:22.7658154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7658250Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7658526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7658608Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7658902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7659022Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7659293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-10-10T00:55:22.7659377Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7659381Z 2025-10-10T00:55:22.7659494Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7659694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7659763Z res = mod(**inputs) 2025-10-10T00:55:22.7660052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7660126Z outputs = self.electra( 2025-10-10T00:55:22.7660408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7660485Z hidden_states = self.encoder( 2025-10-10T00:55:22.7660769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7660847Z layer_outputs = layer_module( 2025-10-10T00:55:22.7661086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7661201Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7661469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7661553Z return func(*args, **kwargs) 2025-10-10T00:55:22.7661839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7661933Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7662245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7662331Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7662712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-10-10T00:55:22.7662842Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:55:22.7663134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-10-10T00:55:22.7663264Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:55:22.7663502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:55:22.7663591Z return self.act(input) 2025-10-10T00:55:22.7663595Z 2025-10-10T00:55:22.7663714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7663941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7664017Z res = mod(**inputs) 2025-10-10T00:55:22.7664312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-10-10T00:55:22.7664401Z outputs = self.electra( 2025-10-10T00:55:22.7664690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-10-10T00:55:22.7664782Z hidden_states = self.encoder( 2025-10-10T00:55:22.7665079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-10-10T00:55:22.7665159Z layer_outputs = layer_module( 2025-10-10T00:55:22.7665403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:22.7665496Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:22.7665764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:22.7665842Z return func(*args, **kwargs) 2025-10-10T00:55:22.7666194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-10-10T00:55:22.7666295Z layer_output = apply_chunking_to_forward( 2025-10-10T00:55:22.7666579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:55:22.7666672Z return forward_fn(*input_tensors) 2025-10-10T00:55:22.7667006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-10-10T00:55:22.7667166Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:55:22.7667457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-10-10T00:55:22.7667549Z hidden_states = self.dense(hidden_states) 2025-10-10T00:55:22.7667564Z 2025-10-10T00:55:22.7667685Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7667906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7668016Z res = mod(**inputs) 2025-10-10T00:55:22.7668308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-10-10T00:55:22.7668520Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-10-10T00:55:22.7668786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 630, in forward 2025-10-10T00:55:22.7668894Z hidden_states = self.dense(generator_hidden_states) 2025-10-10T00:55:22.7668898Z 2025-10-10T00:55:22.7669013Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7669215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7669343Z res = mod(**inputs) 2025-10-10T00:55:22.7669612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-10-10T00:55:22.7669794Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-10-10T00:55:22.7669805Z 2025-10-10T00:55:22.7669909Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:22.7670107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:22.7670182Z res = mod(**inputs) 2025-10-10T00:55:22.7670450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1554, in forward 2025-10-10T00:55:22.7670532Z lm_loss = self.loss_function( 2025-10-10T00:55:22.7670787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T00:55:22.7670967Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T00:55:22.7671225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T00:55:22.7671426Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T00:55:22.7671430Z 2025-10-10T00:55:33.1080980Z Compilation time (from dynamo_timed): 17.748371066 2025-10-10T00:55:33.1166883Z pass 2025-10-10T00:55:33.1167417Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:33.1168319Z TIMING: _recursive_pre_grad_passes:0.00753 _recursive_joint_graph_passes:0.48604 _recursive_post_grad_passes:0.07329 async_compile.wait:0.75723 code_gen:9.78243 inductor_compile:11.10691 backend_compile:14.78211 gc:0.00142 entire_frame_compile:17.74837 total_wall_time:17.74837 2025-10-10T00:55:33.1169385Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8773 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:2529 2025-10-10T00:55:33.1169937Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-10-10T00:55:35.9192134Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:55:35.9195822Z import pynvml # type: ignore[import] 2025-10-10T00:55:39.4251746Z 2025-10-10T00:55:40.8420428Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:55:40.8421311Z loading model: 0it [00:01, ?it/s] 2025-10-10T00:55:40.8425218Z cpu eval GPT2ForSequenceClassification 2025-10-10T00:55:41.6168685Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:41.9918013Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:42.3580538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:55:49.6806611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6807128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6807499Z res = mod(**inputs) 2025-10-10T00:55:49.6807935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6808396Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6808852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:55:49.6809304Z causal_mask = create_causal_mask( 2025-10-10T00:55:49.6810126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:55:49.6810813Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:55:49.6811398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:55:49.6811917Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:55:49.6812449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 654, in find_packed_sequence_indices 2025-10-10T00:55:49.6813004Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-10-10T00:55:49.6813256Z 2025-10-10T00:55:49.6813353Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6813623Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6814029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6814417Z res = mod(**inputs) 2025-10-10T00:55:49.6814815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1396, in forward 2025-10-10T00:55:49.6815294Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-10-10T00:55:49.6815484Z 2025-10-10T00:55:49.6815609Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6816037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6816394Z res = mod(**inputs) 2025-10-10T00:55:49.6816792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6817228Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6817675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6818097Z outputs = block( 2025-10-10T00:55:49.6818463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6818887Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6819322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6819744Z return func(*args, **kwargs) 2025-10-10T00:55:49.6820150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6820591Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6821028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6821456Z return func(*args, **kwargs) 2025-10-10T00:55:49.6821876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.6822442Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.6823041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6823504Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6823701Z 2025-10-10T00:55:49.6823827Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6824244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6824614Z res = mod(**inputs) 2025-10-10T00:55:49.6825021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6825464Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6825944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:55:49.6826582Z causal_mask = create_causal_mask( 2025-10-10T00:55:49.6827003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:55:49.6827576Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:55:49.6828155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:55:49.6828671Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:55:49.6829184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-10-10T00:55:49.6829718Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-10-10T00:55:49.6829973Z 2025-10-10T00:55:49.6830074Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6830347Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6830920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6831295Z res = mod(**inputs) 2025-10-10T00:55:49.6832052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6832507Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6832944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:55:49.6833381Z causal_mask = create_causal_mask( 2025-10-10T00:55:49.6833801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:55:49.6834358Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:55:49.6834948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:55:49.6835466Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:55:49.6835975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-10-10T00:55:49.6836446Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-10-10T00:55:49.6836636Z 2025-10-10T00:55:49.6836752Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6837140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6837503Z res = mod(**inputs) 2025-10-10T00:55:49.6837868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6838269Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6838669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-10-10T00:55:49.6839077Z causal_mask = create_causal_mask( 2025-10-10T00:55:49.6839535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-10-10T00:55:49.6840083Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-10-10T00:55:49.6840641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-10-10T00:55:49.6841109Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-10-10T00:55:49.6841569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-10-10T00:55:49.6842076Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-10-10T00:55:49.6842270Z 2025-10-10T00:55:49.6842363Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6842612Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6843017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6843383Z res = mod(**inputs) 2025-10-10T00:55:49.6843770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6844200Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6844625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6845036Z outputs = block( 2025-10-10T00:55:49.6845393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6845793Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6846189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6846566Z return func(*args, **kwargs) 2025-10-10T00:55:49.6846950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6847357Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6847760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6848162Z return func(*args, **kwargs) 2025-10-10T00:55:49.6848557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.6848973Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.6849436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.6849935Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.6850122Z 2025-10-10T00:55:49.6850234Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6850604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6850938Z res = mod(**inputs) 2025-10-10T00:55:49.6851308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6851732Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6852153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6852533Z outputs = block( 2025-10-10T00:55:49.6852883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6853295Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6853684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6854078Z return func(*args, **kwargs) 2025-10-10T00:55:49.6854460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6854866Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6855271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6855668Z return func(*args, **kwargs) 2025-10-10T00:55:49.6856069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.6856491Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.6856915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6857367Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6857557Z 2025-10-10T00:55:49.6857674Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6858068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6858417Z res = mod(**inputs) 2025-10-10T00:55:49.6858807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6859233Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6859648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6860056Z outputs = block( 2025-10-10T00:55:49.6860414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6860814Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6861221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6861630Z return func(*args, **kwargs) 2025-10-10T00:55:49.6862034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6862486Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6862933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.6863352Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.6863747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6864180Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6864372Z 2025-10-10T00:55:49.6864500Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6864902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6865264Z res = mod(**inputs) 2025-10-10T00:55:49.6865654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6866082Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6866685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6867091Z outputs = block( 2025-10-10T00:55:49.6867448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6867846Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6868266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6868684Z return func(*args, **kwargs) 2025-10-10T00:55:49.6869083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6869564Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6870014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.6870441Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.6870821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.6871335Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.6871597Z 2025-10-10T00:55:49.6871714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6872161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6872520Z res = mod(**inputs) 2025-10-10T00:55:49.6872901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6873305Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6873700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6874082Z outputs = block( 2025-10-10T00:55:49.6874416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6874781Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6875170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6875554Z return func(*args, **kwargs) 2025-10-10T00:55:49.6875940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6876361Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6876772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.6877180Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.6877555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6877969Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6878144Z 2025-10-10T00:55:49.6878253Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6878630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6878961Z res = mod(**inputs) 2025-10-10T00:55:49.6879335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6879740Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6880131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6880514Z outputs = block( 2025-10-10T00:55:49.6880850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6881227Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6881620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6881996Z return func(*args, **kwargs) 2025-10-10T00:55:49.6882375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6882786Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6883185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6883582Z return func(*args, **kwargs) 2025-10-10T00:55:49.6883960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.6884464Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.6884938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6885347Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6885521Z 2025-10-10T00:55:49.6885607Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6885858Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6886277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6886625Z res = mod(**inputs) 2025-10-10T00:55:49.6886997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6887398Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6887802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6888184Z outputs = block( 2025-10-10T00:55:49.6888520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6888889Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6889282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6889668Z return func(*args, **kwargs) 2025-10-10T00:55:49.6890044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6890448Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6890830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6891205Z return func(*args, **kwargs) 2025-10-10T00:55:49.6891576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.6891987Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.6892438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.6892937Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.6893132Z 2025-10-10T00:55:49.6893240Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6893622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6893953Z res = mod(**inputs) 2025-10-10T00:55:49.6894319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6894729Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6895139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6895514Z outputs = block( 2025-10-10T00:55:49.6895844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6896206Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6896599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6896988Z return func(*args, **kwargs) 2025-10-10T00:55:49.6897373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6897800Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6898189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6898572Z return func(*args, **kwargs) 2025-10-10T00:55:49.6898954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.6899340Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.6899689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6900090Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6900272Z 2025-10-10T00:55:49.6900416Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6900802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6901132Z res = mod(**inputs) 2025-10-10T00:55:49.6901496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6901893Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6902288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6902666Z outputs = block( 2025-10-10T00:55:49.6902996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6903393Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6903800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6904208Z return func(*args, **kwargs) 2025-10-10T00:55:49.6904608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6905049Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6905489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.6905911Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.6906384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6906827Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6907031Z 2025-10-10T00:55:49.6907145Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6907542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6907911Z res = mod(**inputs) 2025-10-10T00:55:49.6908300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6908720Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6909142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6909548Z outputs = block( 2025-10-10T00:55:49.6909903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6910299Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6910708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6911121Z return func(*args, **kwargs) 2025-10-10T00:55:49.6911525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6911972Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6912420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.6912858Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.6913239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.6913741Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.6913988Z 2025-10-10T00:55:49.6914111Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6914519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6914870Z res = mod(**inputs) 2025-10-10T00:55:49.6915296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6915746Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6916166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6916560Z outputs = block( 2025-10-10T00:55:49.6916895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6917267Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6917655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6918053Z return func(*args, **kwargs) 2025-10-10T00:55:49.6918426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6918850Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6919272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.6919679Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.6920055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6920457Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6920642Z 2025-10-10T00:55:49.6920751Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6921124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6921456Z res = mod(**inputs) 2025-10-10T00:55:49.6921814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6922223Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6922621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6923017Z outputs = block( 2025-10-10T00:55:49.6923371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6923765Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6924181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6924592Z return func(*args, **kwargs) 2025-10-10T00:55:49.6924971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6925376Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6925768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6926148Z return func(*args, **kwargs) 2025-10-10T00:55:49.6926531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.6927033Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.6927514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6927927Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6928108Z 2025-10-10T00:55:49.6928194Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6928444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6928811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6929136Z res = mod(**inputs) 2025-10-10T00:55:49.6929536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6929959Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6930364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6930753Z outputs = block( 2025-10-10T00:55:49.6931085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6931686Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6932082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6932462Z return func(*args, **kwargs) 2025-10-10T00:55:49.6932832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6933232Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6933631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6934020Z return func(*args, **kwargs) 2025-10-10T00:55:49.6934404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.6934828Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.6935280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.6935764Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.6935945Z 2025-10-10T00:55:49.6936060Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6936420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6936733Z res = mod(**inputs) 2025-10-10T00:55:49.6937094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6937493Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6937882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6938252Z outputs = block( 2025-10-10T00:55:49.6938579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6938946Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6939326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6939695Z return func(*args, **kwargs) 2025-10-10T00:55:49.6940060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6940458Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6940853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6941235Z return func(*args, **kwargs) 2025-10-10T00:55:49.6941699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.6942107Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.6942478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6942892Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6943071Z 2025-10-10T00:55:49.6943191Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6943574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6943933Z res = mod(**inputs) 2025-10-10T00:55:49.6944407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6944834Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6945254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6945626Z outputs = block( 2025-10-10T00:55:49.6945958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6946393Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6946803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6947205Z return func(*args, **kwargs) 2025-10-10T00:55:49.6947613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6948042Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6948483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.6948908Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.6949287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6949787Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6949983Z 2025-10-10T00:55:49.6950099Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6950499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6950865Z res = mod(**inputs) 2025-10-10T00:55:49.6951262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6951705Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6952134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6952541Z outputs = block( 2025-10-10T00:55:49.6952893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6953291Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6953713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6954130Z return func(*args, **kwargs) 2025-10-10T00:55:49.6954536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6954989Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6955399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.6955794Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.6956151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.6956627Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.6956856Z 2025-10-10T00:55:49.6956962Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6957331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6957667Z res = mod(**inputs) 2025-10-10T00:55:49.6958043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6958454Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6958895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6959300Z outputs = block( 2025-10-10T00:55:49.6959622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6959981Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6960348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6960717Z return func(*args, **kwargs) 2025-10-10T00:55:49.6961083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6961487Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6961892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.6962282Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.6962662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6963053Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6963225Z 2025-10-10T00:55:49.6963336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6963694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6964038Z res = mod(**inputs) 2025-10-10T00:55:49.6964422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6964862Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6965278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6965668Z outputs = block( 2025-10-10T00:55:49.6966014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6966401Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6966778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6967152Z return func(*args, **kwargs) 2025-10-10T00:55:49.6967512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:55:49.6967933Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:55:49.6968106Z 2025-10-10T00:55:49.6968216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6968607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6968946Z res = mod(**inputs) 2025-10-10T00:55:49.6969332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6969753Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6970149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6970552Z outputs = block( 2025-10-10T00:55:49.6970879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6971252Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6971635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6972017Z return func(*args, **kwargs) 2025-10-10T00:55:49.6972393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6972808Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6973260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6973692Z return func(*args, **kwargs) 2025-10-10T00:55:49.6974091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.6974621Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.6975095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6975499Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6975684Z 2025-10-10T00:55:49.6975772Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.6976023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6976387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6976714Z res = mod(**inputs) 2025-10-10T00:55:49.6977086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6977489Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6977890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6978262Z outputs = block( 2025-10-10T00:55:49.6978592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6978962Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6979348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6979754Z return func(*args, **kwargs) 2025-10-10T00:55:49.6980154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6980594Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6980984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6981367Z return func(*args, **kwargs) 2025-10-10T00:55:49.6981736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.6982148Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.6982601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.6983091Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.6983274Z 2025-10-10T00:55:49.6983388Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6983761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6984112Z res = mod(**inputs) 2025-10-10T00:55:49.6984500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6984948Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6985360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6985764Z outputs = block( 2025-10-10T00:55:49.6986118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6986637Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6987067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6987528Z return func(*args, **kwargs) 2025-10-10T00:55:49.6987947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.6988406Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.6988888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6989316Z return func(*args, **kwargs) 2025-10-10T00:55:49.6989734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.6990168Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.6990565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6991016Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6991220Z 2025-10-10T00:55:49.6991337Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6991758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6992116Z res = mod(**inputs) 2025-10-10T00:55:49.6992536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6992987Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.6993409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.6993822Z outputs = block( 2025-10-10T00:55:49.6994183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.6994585Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.6995066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.6995478Z return func(*args, **kwargs) 2025-10-10T00:55:49.6995867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.6996353Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.6996770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.6997160Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.6997527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.6997928Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.6998102Z 2025-10-10T00:55:49.6998217Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.6998587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.6998928Z res = mod(**inputs) 2025-10-10T00:55:49.6999315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.6999739Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7000154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7000585Z outputs = block( 2025-10-10T00:55:49.7000930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7001319Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7001726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7002137Z return func(*args, **kwargs) 2025-10-10T00:55:49.7002526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7002969Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7003462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7003883Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7004267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7004750Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7005009Z 2025-10-10T00:55:49.7005126Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7005519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7005866Z res = mod(**inputs) 2025-10-10T00:55:49.7006258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7006673Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7007099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7007497Z outputs = block( 2025-10-10T00:55:49.7007845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7008227Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7008635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7009033Z return func(*args, **kwargs) 2025-10-10T00:55:49.7009431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7009870Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7010300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7010731Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7011121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7011550Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7011735Z 2025-10-10T00:55:49.7011856Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7012237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7012581Z res = mod(**inputs) 2025-10-10T00:55:49.7012965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7013389Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7013796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7014202Z outputs = block( 2025-10-10T00:55:49.7014552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7014944Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7015375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7015771Z return func(*args, **kwargs) 2025-10-10T00:55:49.7016170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7016598Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7017019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7017420Z return func(*args, **kwargs) 2025-10-10T00:55:49.7017844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7018393Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7018890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7019320Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7019503Z 2025-10-10T00:55:49.7019600Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7019854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7020245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7020594Z res = mod(**inputs) 2025-10-10T00:55:49.7020983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7021427Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7021874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7022278Z outputs = block( 2025-10-10T00:55:49.7022631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7023030Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7023443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7023859Z return func(*args, **kwargs) 2025-10-10T00:55:49.7024269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7024707Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7025130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7025551Z return func(*args, **kwargs) 2025-10-10T00:55:49.7025960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7026494Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7027007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7027544Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7027761Z 2025-10-10T00:55:49.7027887Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7028281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7028661Z res = mod(**inputs) 2025-10-10T00:55:49.7029064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7029507Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7029947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7030390Z outputs = block( 2025-10-10T00:55:49.7030755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7031229Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7031872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7032304Z return func(*args, **kwargs) 2025-10-10T00:55:49.7032722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7033166Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7033680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7034130Z return func(*args, **kwargs) 2025-10-10T00:55:49.7034545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7034982Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7035385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7035824Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7036027Z 2025-10-10T00:55:49.7036150Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7036566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7036937Z res = mod(**inputs) 2025-10-10T00:55:49.7037332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7037787Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7038227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7038648Z outputs = block( 2025-10-10T00:55:49.7039013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7039412Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7039837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7040263Z return func(*args, **kwargs) 2025-10-10T00:55:49.7040661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7041083Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7041499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7041900Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7042273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7042685Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7042862Z 2025-10-10T00:55:49.7042971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7043345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7043676Z res = mod(**inputs) 2025-10-10T00:55:49.7044044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7044445Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7044836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7045222Z outputs = block( 2025-10-10T00:55:49.7045553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7045951Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7046331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7046712Z return func(*args, **kwargs) 2025-10-10T00:55:49.7047088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7047508Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7047924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7048309Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7048733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7049198Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7049434Z 2025-10-10T00:55:49.7049550Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7049940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7050293Z res = mod(**inputs) 2025-10-10T00:55:49.7050664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7051076Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7051497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7051888Z outputs = block( 2025-10-10T00:55:49.7052214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7052588Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7052974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7053356Z return func(*args, **kwargs) 2025-10-10T00:55:49.7053727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7054150Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7054589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7055009Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7055380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7055804Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7055998Z 2025-10-10T00:55:49.7056112Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7056505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7056857Z res = mod(**inputs) 2025-10-10T00:55:49.7057248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7057643Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7058039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7058424Z outputs = block( 2025-10-10T00:55:49.7058775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7059159Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7059573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7059975Z return func(*args, **kwargs) 2025-10-10T00:55:49.7060404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:55:49.7060858Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:55:49.7061030Z 2025-10-10T00:55:49.7061144Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7061536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7061886Z res = mod(**inputs) 2025-10-10T00:55:49.7062274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7062696Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7063195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7063639Z outputs = block( 2025-10-10T00:55:49.7064011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7064404Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7064806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7065222Z return func(*args, **kwargs) 2025-10-10T00:55:49.7065626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7066056Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7066558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7066972Z return func(*args, **kwargs) 2025-10-10T00:55:49.7067390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7067940Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7068456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7068905Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7069092Z 2025-10-10T00:55:49.7069185Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7069456Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7069847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7070195Z res = mod(**inputs) 2025-10-10T00:55:49.7070577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7071007Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7071427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7071828Z outputs = block( 2025-10-10T00:55:49.7072178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7072561Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7072967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7073368Z return func(*args, **kwargs) 2025-10-10T00:55:49.7073765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7074185Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7074606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7075013Z return func(*args, **kwargs) 2025-10-10T00:55:49.7075404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7075849Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7076298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7076787Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7076981Z 2025-10-10T00:55:49.7077088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7077458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7077788Z res = mod(**inputs) 2025-10-10T00:55:49.7078184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7078607Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7079008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7079391Z outputs = block( 2025-10-10T00:55:49.7079715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7080085Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7080470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7080849Z return func(*args, **kwargs) 2025-10-10T00:55:49.7081226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7081622Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7082020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7082401Z return func(*args, **kwargs) 2025-10-10T00:55:49.7082776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7083174Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7083531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7083936Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7084120Z 2025-10-10T00:55:49.7084230Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7084601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7084922Z res = mod(**inputs) 2025-10-10T00:55:49.7085291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7085693Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7086091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7086472Z outputs = block( 2025-10-10T00:55:49.7086801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7087193Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7087602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7088012Z return func(*args, **kwargs) 2025-10-10T00:55:49.7088411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7088850Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7089300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7089718Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7090081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7090482Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7090664Z 2025-10-10T00:55:49.7090772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7091136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7091462Z res = mod(**inputs) 2025-10-10T00:55:49.7091826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7092218Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7092672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7093053Z outputs = block( 2025-10-10T00:55:49.7093387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7093757Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7094138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7094523Z return func(*args, **kwargs) 2025-10-10T00:55:49.7094901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7095322Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7095729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7096139Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7096521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7097026Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7097275Z 2025-10-10T00:55:49.7097397Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7097798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7098163Z res = mod(**inputs) 2025-10-10T00:55:49.7098549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7098971Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7099389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7099786Z outputs = block( 2025-10-10T00:55:49.7100136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7100526Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7100938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7101353Z return func(*args, **kwargs) 2025-10-10T00:55:49.7101768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7102221Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7102673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7103111Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7103510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7103959Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7104182Z 2025-10-10T00:55:49.7104299Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7104700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7105059Z res = mod(**inputs) 2025-10-10T00:55:49.7105453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7105898Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7106408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7106836Z outputs = block( 2025-10-10T00:55:49.7108181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7108611Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7109022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7109442Z return func(*args, **kwargs) 2025-10-10T00:55:49.7109844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7110319Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7110739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7111147Z return func(*args, **kwargs) 2025-10-10T00:55:49.7111546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7112083Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7112577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7113010Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7113202Z 2025-10-10T00:55:49.7113291Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7113555Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7113949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7114293Z res = mod(**inputs) 2025-10-10T00:55:49.7114687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7115116Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7115540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7115944Z outputs = block( 2025-10-10T00:55:49.7116302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7116703Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7117118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7117522Z return func(*args, **kwargs) 2025-10-10T00:55:49.7117915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7118352Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7118749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7119132Z return func(*args, **kwargs) 2025-10-10T00:55:49.7119507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7119926Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7120403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7120945Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7121143Z 2025-10-10T00:55:49.7121264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7121650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7122000Z res = mod(**inputs) 2025-10-10T00:55:49.7122381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7122785Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7123218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7123606Z outputs = block( 2025-10-10T00:55:49.7123943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7124324Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7124716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7125097Z return func(*args, **kwargs) 2025-10-10T00:55:49.7125481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7125904Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7126334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7126716Z return func(*args, **kwargs) 2025-10-10T00:55:49.7127099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7127500Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7127872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7128291Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7128477Z 2025-10-10T00:55:49.7128599Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7128990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7129342Z res = mod(**inputs) 2025-10-10T00:55:49.7129731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7130167Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7130570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7130977Z outputs = block( 2025-10-10T00:55:49.7131334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7131916Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7132331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7132739Z return func(*args, **kwargs) 2025-10-10T00:55:49.7133214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7133661Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7134107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7134532Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7134922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7135358Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7135611Z 2025-10-10T00:55:49.7135728Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7136127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7136485Z res = mod(**inputs) 2025-10-10T00:55:49.7136890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7137321Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7137758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7138215Z outputs = block( 2025-10-10T00:55:49.7138629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7139055Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7139471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7139886Z return func(*args, **kwargs) 2025-10-10T00:55:49.7140287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7140724Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7141165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7141586Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7141969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7142459Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7142715Z 2025-10-10T00:55:49.7142832Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7143237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7143585Z res = mod(**inputs) 2025-10-10T00:55:49.7143972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7144400Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7144820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7145220Z outputs = block( 2025-10-10T00:55:49.7145569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7145965Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7146426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7146844Z return func(*args, **kwargs) 2025-10-10T00:55:49.7147242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7147682Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7148112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7148536Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7148925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7149365Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7149551Z 2025-10-10T00:55:49.7149672Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7150034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7150372Z res = mod(**inputs) 2025-10-10T00:55:49.7150755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7151149Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7151536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7151899Z outputs = block( 2025-10-10T00:55:49.7152222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7152585Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7152961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7153373Z return func(*args, **kwargs) 2025-10-10T00:55:49.7153749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:55:49.7154170Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:55:49.7154340Z 2025-10-10T00:55:49.7154464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7154851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7155191Z res = mod(**inputs) 2025-10-10T00:55:49.7155585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7156000Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7156404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7156771Z outputs = block( 2025-10-10T00:55:49.7157092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7157454Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7157831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7158199Z return func(*args, **kwargs) 2025-10-10T00:55:49.7158560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7158955Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7159345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7159724Z return func(*args, **kwargs) 2025-10-10T00:55:49.7160102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7160590Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7161053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7161450Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7161620Z 2025-10-10T00:55:49.7161710Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7161951Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7162304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7162625Z res = mod(**inputs) 2025-10-10T00:55:49.7162983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7163376Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7163769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7164149Z outputs = block( 2025-10-10T00:55:49.7164481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7164871Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7165260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7165632Z return func(*args, **kwargs) 2025-10-10T00:55:49.7166009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7166417Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7166816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7167228Z return func(*args, **kwargs) 2025-10-10T00:55:49.7167630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7168048Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7168507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7168998Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7169183Z 2025-10-10T00:55:49.7169290Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7169661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7169992Z res = mod(**inputs) 2025-10-10T00:55:49.7170356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7170760Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7171154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7171532Z outputs = block( 2025-10-10T00:55:49.7171864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7172237Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7172621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7172993Z return func(*args, **kwargs) 2025-10-10T00:55:49.7173370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7173770Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7174162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7174537Z return func(*args, **kwargs) 2025-10-10T00:55:49.7174912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7175310Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7175673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7176075Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7176249Z 2025-10-10T00:55:49.7176358Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7176725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7177049Z res = mod(**inputs) 2025-10-10T00:55:49.7177415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7177840Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7178258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7178688Z outputs = block( 2025-10-10T00:55:49.7179018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7179387Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7179765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7180144Z return func(*args, **kwargs) 2025-10-10T00:55:49.7180519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7180938Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7181383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7181792Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7182157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7182569Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7182751Z 2025-10-10T00:55:49.7182870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7183258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7183599Z res = mod(**inputs) 2025-10-10T00:55:49.7183985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7184406Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7184829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7185239Z outputs = block( 2025-10-10T00:55:49.7185601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7186004Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7186507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7186946Z return func(*args, **kwargs) 2025-10-10T00:55:49.7187358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7187848Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7188293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7188736Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7189120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7189613Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7189878Z 2025-10-10T00:55:49.7189994Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7190389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7190741Z res = mod(**inputs) 2025-10-10T00:55:49.7191126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7191555Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7191980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7192390Z outputs = block( 2025-10-10T00:55:49.7192747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7193140Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7193553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7193980Z return func(*args, **kwargs) 2025-10-10T00:55:49.7194380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7194829Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7195264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7195696Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7196094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7196582Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7196770Z 2025-10-10T00:55:49.7196886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7197275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7197623Z res = mod(**inputs) 2025-10-10T00:55:49.7198006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7198421Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7198831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7199229Z outputs = block( 2025-10-10T00:55:49.7199574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7199963Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7200352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7200721Z return func(*args, **kwargs) 2025-10-10T00:55:49.7201095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7201493Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7201885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7202260Z return func(*args, **kwargs) 2025-10-10T00:55:49.7202632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7203144Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7203641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7204072Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7204257Z 2025-10-10T00:55:49.7204348Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7204613Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7205001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7205347Z res = mod(**inputs) 2025-10-10T00:55:49.7205735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7206148Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7206564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7206962Z outputs = block( 2025-10-10T00:55:49.7207317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7207700Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7208111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7208531Z return func(*args, **kwargs) 2025-10-10T00:55:49.7208929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7209367Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7209780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7210189Z return func(*args, **kwargs) 2025-10-10T00:55:49.7210583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7211018Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7211553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7212044Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7212239Z 2025-10-10T00:55:49.7212478Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7212873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7213226Z res = mod(**inputs) 2025-10-10T00:55:49.7213608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7214036Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7214461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7214864Z outputs = block( 2025-10-10T00:55:49.7215226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7215614Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7216009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7216390Z return func(*args, **kwargs) 2025-10-10T00:55:49.7216787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7217217Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7217633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7218031Z return func(*args, **kwargs) 2025-10-10T00:55:49.7218428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7218523Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7218772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7218903Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7218907Z 2025-10-10T00:55:49.7219026Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7219242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7219311Z res = mod(**inputs) 2025-10-10T00:55:49.7219589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7219678Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7219951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7220021Z outputs = block( 2025-10-10T00:55:49.7220265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7220359Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7220648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7220732Z return func(*args, **kwargs) 2025-10-10T00:55:49.7220996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7221118Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7221384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7221474Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7221716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7221896Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7221900Z 2025-10-10T00:55:49.7222022Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7222236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7222307Z res = mod(**inputs) 2025-10-10T00:55:49.7222581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7222671Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7222941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7223011Z outputs = block( 2025-10-10T00:55:49.7223249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7223347Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7223604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7223692Z return func(*args, **kwargs) 2025-10-10T00:55:49.7223959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7224078Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7224342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7224430Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7224669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7224861Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7224866Z 2025-10-10T00:55:49.7224989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7225202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7225274Z res = mod(**inputs) 2025-10-10T00:55:49.7225557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7225650Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7225928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7226001Z outputs = block( 2025-10-10T00:55:49.7226315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7226414Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7226688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7226781Z return func(*args, **kwargs) 2025-10-10T00:55:49.7227060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7227212Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7227493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7227587Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7227834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7227965Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7227969Z 2025-10-10T00:55:49.7228094Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7228350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7228450Z res = mod(**inputs) 2025-10-10T00:55:49.7228717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7228808Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7229078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7229148Z outputs = block( 2025-10-10T00:55:49.7229391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7229478Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7229733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7229816Z return func(*args, **kwargs) 2025-10-10T00:55:49.7230084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:55:49.7230208Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:55:49.7230212Z 2025-10-10T00:55:49.7230325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7230543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7230614Z res = mod(**inputs) 2025-10-10T00:55:49.7230879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7230976Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7231239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7231313Z outputs = block( 2025-10-10T00:55:49.7231772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7231865Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7232131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7232209Z return func(*args, **kwargs) 2025-10-10T00:55:49.7232482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7232578Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7232836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7232921Z return func(*args, **kwargs) 2025-10-10T00:55:49.7233187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7233396Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7233638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7233774Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7233824Z 2025-10-10T00:55:49.7233916Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7234027Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7234249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7234319Z res = mod(**inputs) 2025-10-10T00:55:49.7234591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7234680Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7234955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7235099Z outputs = block( 2025-10-10T00:55:49.7235326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7235417Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7235664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7235735Z return func(*args, **kwargs) 2025-10-10T00:55:49.7235994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7236085Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7236336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7236405Z return func(*args, **kwargs) 2025-10-10T00:55:49.7236665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7236767Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7237068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7237211Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7237216Z 2025-10-10T00:55:49.7237319Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7237530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7237597Z res = mod(**inputs) 2025-10-10T00:55:49.7237852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7237943Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7238196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7238272Z outputs = block( 2025-10-10T00:55:49.7238498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7238587Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7238832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7238903Z return func(*args, **kwargs) 2025-10-10T00:55:49.7239161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7239250Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7239503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7239573Z return func(*args, **kwargs) 2025-10-10T00:55:49.7239825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7239921Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7240142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7240288Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7240291Z 2025-10-10T00:55:49.7240396Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7240606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7240671Z res = mod(**inputs) 2025-10-10T00:55:49.7240923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7241012Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7241295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7241394Z outputs = block( 2025-10-10T00:55:49.7241617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7241700Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7241950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7242022Z return func(*args, **kwargs) 2025-10-10T00:55:49.7242281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7242388Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7242639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7242729Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7242954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7243080Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7243085Z 2025-10-10T00:55:49.7243190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7243399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7243466Z res = mod(**inputs) 2025-10-10T00:55:49.7243717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7243810Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7244065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7244137Z outputs = block( 2025-10-10T00:55:49.7244366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7244448Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7244702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7244774Z return func(*args, **kwargs) 2025-10-10T00:55:49.7245032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7245137Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7245388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7245478Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7245697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7245887Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7245892Z 2025-10-10T00:55:49.7245998Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7246224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7246292Z res = mod(**inputs) 2025-10-10T00:55:49.7246550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7246653Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7246901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7246973Z outputs = block( 2025-10-10T00:55:49.7247193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7247273Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7247562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7247634Z return func(*args, **kwargs) 2025-10-10T00:55:49.7247887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7247989Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7248238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7248325Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7248544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7248667Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7248671Z 2025-10-10T00:55:49.7248775Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7248980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7249044Z res = mod(**inputs) 2025-10-10T00:55:49.7249296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7249384Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7249630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7249701Z outputs = block( 2025-10-10T00:55:49.7249921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7250007Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7250247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7250320Z return func(*args, **kwargs) 2025-10-10T00:55:49.7250573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7250661Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7250911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7250980Z return func(*args, **kwargs) 2025-10-10T00:55:49.7251225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7251418Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7251636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7251760Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7251769Z 2025-10-10T00:55:49.7251851Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7251960Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7252173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7252238Z res = mod(**inputs) 2025-10-10T00:55:49.7252491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7252572Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7252820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7252883Z outputs = block( 2025-10-10T00:55:49.7253100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7253189Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7253504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7253582Z return func(*args, **kwargs) 2025-10-10T00:55:49.7253825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7253914Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7254166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7254240Z return func(*args, **kwargs) 2025-10-10T00:55:49.7254496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7254596Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7254904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7255035Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7255039Z 2025-10-10T00:55:49.7255140Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7255345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7255411Z res = mod(**inputs) 2025-10-10T00:55:49.7255664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7255745Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7255992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7256067Z outputs = block( 2025-10-10T00:55:49.7256290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7256386Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7256633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7256706Z return func(*args, **kwargs) 2025-10-10T00:55:49.7256965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7257053Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7257304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7257374Z return func(*args, **kwargs) 2025-10-10T00:55:49.7257629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7257713Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7257941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7258064Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7258068Z 2025-10-10T00:55:49.7258187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7258386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7258450Z res = mod(**inputs) 2025-10-10T00:55:49.7258695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7258783Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7259024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7259095Z outputs = block( 2025-10-10T00:55:49.7259313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7259443Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7259682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7259752Z return func(*args, **kwargs) 2025-10-10T00:55:49.7260007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7260110Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7260368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7260451Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7260672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7260799Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7260804Z 2025-10-10T00:55:49.7260912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7261122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7261189Z res = mod(**inputs) 2025-10-10T00:55:49.7261448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7261546Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7261812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7261889Z outputs = block( 2025-10-10T00:55:49.7262129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7262217Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7262465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7262536Z return func(*args, **kwargs) 2025-10-10T00:55:49.7262804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7262919Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7263192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7263279Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7263511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7263710Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7263714Z 2025-10-10T00:55:49.7263825Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7264052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7264123Z res = mod(**inputs) 2025-10-10T00:55:49.7264403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7264512Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7264778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7264853Z outputs = block( 2025-10-10T00:55:49.7265092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7265184Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7265453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7265527Z return func(*args, **kwargs) 2025-10-10T00:55:49.7265897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7266010Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7266345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7266447Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7266687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7266813Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7266818Z 2025-10-10T00:55:49.7266929Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7267158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7267232Z res = mod(**inputs) 2025-10-10T00:55:49.7267524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7267623Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7267888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7267968Z outputs = block( 2025-10-10T00:55:49.7268203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7268297Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7268563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7268639Z return func(*args, **kwargs) 2025-10-10T00:55:49.7268910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-10-10T00:55:49.7269030Z hidden_states = residual + feed_forward_hidden_states 2025-10-10T00:55:49.7269034Z 2025-10-10T00:55:49.7269155Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7269366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7269447Z res = mod(**inputs) 2025-10-10T00:55:49.7269714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7269804Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7270076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7270146Z outputs = block( 2025-10-10T00:55:49.7270387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7270471Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7270730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7270815Z return func(*args, **kwargs) 2025-10-10T00:55:49.7271078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7271206Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7271468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7271554Z return func(*args, **kwargs) 2025-10-10T00:55:49.7271820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-10-10T00:55:49.7272025Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-10-10T00:55:49.7272301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7272464Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7272468Z 2025-10-10T00:55:49.7272567Z cudagraph partition due to non gpu ops 2025-10-10T00:55:49.7272679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7272894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7272972Z res = mod(**inputs) 2025-10-10T00:55:49.7273243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7273339Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7273605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7273672Z outputs = block( 2025-10-10T00:55:49.7273921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7274010Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7274277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7274355Z return func(*args, **kwargs) 2025-10-10T00:55:49.7274628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7274722Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7274981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7275063Z return func(*args, **kwargs) 2025-10-10T00:55:49.7275326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-10-10T00:55:49.7275438Z attn_output, attn_weights = attention_interface( 2025-10-10T00:55:49.7275758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:55:49.7275900Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:55:49.7275913Z 2025-10-10T00:55:49.7276025Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7276239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7276317Z res = mod(**inputs) 2025-10-10T00:55:49.7276587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7276682Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7276947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7277017Z outputs = block( 2025-10-10T00:55:49.7277268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7277354Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7277621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7277717Z return func(*args, **kwargs) 2025-10-10T00:55:49.7277983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-10-10T00:55:49.7278085Z attn_output, self_attn_weights = self.attn( 2025-10-10T00:55:49.7278344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7278426Z return func(*args, **kwargs) 2025-10-10T00:55:49.7278694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-10-10T00:55:49.7278833Z attn_output = self.c_proj(attn_output) 2025-10-10T00:55:49.7279075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7279203Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7279206Z 2025-10-10T00:55:49.7279323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7279546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7279622Z res = mod(**inputs) 2025-10-10T00:55:49.7279892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7279978Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7280248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7280316Z outputs = block( 2025-10-10T00:55:49.7280562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7280648Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7280906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7280987Z return func(*args, **kwargs) 2025-10-10T00:55:49.7281249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7281366Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7281632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-10-10T00:55:49.7281725Z hidden_states = self.c_fc(hidden_states) 2025-10-10T00:55:49.7281965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7282096Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7282100Z 2025-10-10T00:55:49.7282218Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7282431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7282510Z res = mod(**inputs) 2025-10-10T00:55:49.7282778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7282867Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7283139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7283207Z outputs = block( 2025-10-10T00:55:49.7283451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7283538Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7283795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7283879Z return func(*args, **kwargs) 2025-10-10T00:55:49.7284160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7284281Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7284546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-10-10T00:55:49.7284639Z hidden_states = self.act(hidden_states) 2025-10-10T00:55:49.7284875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:55:49.7285063Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:55:49.7285088Z 2025-10-10T00:55:49.7285242Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7285456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7285536Z res = mod(**inputs) 2025-10-10T00:55:49.7285804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-10-10T00:55:49.7285893Z transformer_outputs = self.transformer( 2025-10-10T00:55:49.7286164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-10-10T00:55:49.7286232Z outputs = block( 2025-10-10T00:55:49.7286476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:55:49.7286561Z return super().__call__(*args, **kwargs) 2025-10-10T00:55:49.7286837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:55:49.7286914Z return func(*args, **kwargs) 2025-10-10T00:55:49.7287179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-10-10T00:55:49.7287297Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-10-10T00:55:49.7287561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-10-10T00:55:49.7287661Z hidden_states = self.c_proj(hidden_states) 2025-10-10T00:55:49.7287893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-10-10T00:55:49.7288018Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-10-10T00:55:49.7288030Z 2025-10-10T00:55:49.7288140Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7288358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7288436Z res = mod(**inputs) 2025-10-10T00:55:49.7288700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1381, in forward 2025-10-10T00:55:49.7288792Z logits = self.score(hidden_states) 2025-10-10T00:55:49.7288796Z 2025-10-10T00:55:49.7288905Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7289125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7289200Z res = mod(**inputs) 2025-10-10T00:55:49.7289469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1424, in forward 2025-10-10T00:55:49.7289633Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-10-10T00:55:49.7289637Z 2025-10-10T00:55:49.7289746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:55:49.7289964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:55:49.7290040Z res = mod(**inputs) 2025-10-10T00:55:49.7290292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1424, in forward 2025-10-10T00:55:49.7290464Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-10-10T00:55:49.7290468Z 2025-10-10T00:56:02.5897292Z Compilation time (from dynamo_timed): 18.816100372 2025-10-10T00:56:02.5897601Z pass 2025-10-10T00:56:02.5897916Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:02.5898955Z TIMING: _recursive_pre_grad_passes:0.01417 _recursive_joint_graph_passes:0.33709 _recursive_post_grad_passes:0.0721 async_compile.wait:0.79191 code_gen:9.62343 inductor_compile:10.84811 backend_compile:14.12113 gc:0.00092 entire_frame_compile:18.8161 total_wall_time:18.8161 2025-10-10T00:56:02.5900315Z STATS: call_* op count: 1126 | FakeTensorMode.__torch_dispatch__:7520 | FakeTensor.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1656 2025-10-10T00:56:02.5900987Z Dynamo produced 2 graphs covering 1126 ops with 0 graph breaks (0 unique) 2025-10-10T00:56:05.3394755Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:56:05.3395665Z import pynvml # type: ignore[import] 2025-10-10T00:56:08.8675982Z 2025-10-10T00:56:09.9862928Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:56:09.9863228Z loading model: 0it [00:01, ?it/s] 2025-10-10T00:56:09.9871632Z cpu eval GoogleFnet 2025-10-10T00:56:10.4285038Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:10.5944532Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:10.7653688Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:16.6168308Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6172728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6175457Z res = mod(**inputs) 2025-10-10T00:56:16.6176010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6176468Z outputs = self.fnet( 2025-10-10T00:56:16.6176914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6177355Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6177810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6178273Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6178681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6179092Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6179522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6179998Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6180445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6180897Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6181357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6181831Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6182025Z 2025-10-10T00:56:16.6182168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6182586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6183275Z res = mod(**inputs) 2025-10-10T00:56:16.6183678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6184107Z outputs = self.fnet( 2025-10-10T00:56:16.6184529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6184957Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6185388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6185836Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6186363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6187072Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6187500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6187967Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6188406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6188841Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6189264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6189729Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6189916Z 2025-10-10T00:56:16.6190039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6190458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6190828Z res = mod(**inputs) 2025-10-10T00:56:16.6191220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6191647Z outputs = self.fnet( 2025-10-10T00:56:16.6192043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6192478Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6192899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6193332Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6193751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6194163Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6194611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6195057Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6195505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6196082Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6196514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6196970Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6197146Z 2025-10-10T00:56:16.6197272Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6197671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6198034Z res = mod(**inputs) 2025-10-10T00:56:16.6198433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6198858Z outputs = self.fnet( 2025-10-10T00:56:16.6199326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6199760Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6200186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6200625Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6201095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6201517Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6201955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6202483Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6202931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6203363Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6203798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6204255Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6204424Z 2025-10-10T00:56:16.6204538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6204929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6205275Z res = mod(**inputs) 2025-10-10T00:56:16.6205663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6206072Z outputs = self.fnet( 2025-10-10T00:56:16.6206466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6206883Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6207295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6207727Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6208120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6208518Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6208937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6209380Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6209823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6210257Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6210677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6211130Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6211308Z 2025-10-10T00:56:16.6211433Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6211816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6212165Z res = mod(**inputs) 2025-10-10T00:56:16.6212551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6212964Z outputs = self.fnet( 2025-10-10T00:56:16.6213350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6213763Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6214175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6214627Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6215026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6215419Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6215832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6216276Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6216714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6217136Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6217606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6218056Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6218235Z 2025-10-10T00:56:16.6218348Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6218742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6219090Z res = mod(**inputs) 2025-10-10T00:56:16.6219469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6219883Z outputs = self.fnet( 2025-10-10T00:56:16.6220265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6220678Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6221089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6221509Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6221918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6222329Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6222756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6223195Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6223643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6224076Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6224510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6224967Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6225141Z 2025-10-10T00:56:16.6225257Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6225656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6226017Z res = mod(**inputs) 2025-10-10T00:56:16.6226412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6226945Z outputs = self.fnet( 2025-10-10T00:56:16.6227347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6227776Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6228207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6228648Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6229060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6229471Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6229925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6230380Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6230830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6231273Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6231883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6232360Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6232537Z 2025-10-10T00:56:16.6232664Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6233212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6233576Z res = mod(**inputs) 2025-10-10T00:56:16.6233978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6234403Z outputs = self.fnet( 2025-10-10T00:56:16.6234799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6235227Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6235652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6236079Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6236458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6236831Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6237223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6237641Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6238055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6238455Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6238850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6239296Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6239470Z 2025-10-10T00:56:16.6239585Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6239976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6240330Z res = mod(**inputs) 2025-10-10T00:56:16.6240689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6241073Z outputs = self.fnet( 2025-10-10T00:56:16.6241439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6241829Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6242213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6242611Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6242989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6243362Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6243757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6244177Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6244577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6245009Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6245401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6245819Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6245977Z 2025-10-10T00:56:16.6246091Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6246453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6246783Z res = mod(**inputs) 2025-10-10T00:56:16.6247148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6247584Z outputs = self.fnet( 2025-10-10T00:56:16.6247948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6248346Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6248753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6249195Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6249608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6250004Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6250448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6250873Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6251286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6251682Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6252076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6252498Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6252658Z 2025-10-10T00:56:16.6252772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6253137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6253460Z res = mod(**inputs) 2025-10-10T00:56:16.6253822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6254209Z outputs = self.fnet( 2025-10-10T00:56:16.6254568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6254961Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6255343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6255781Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6256179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6256581Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6256991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6257437Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6257846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6258245Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6258657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6259091Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6259291Z 2025-10-10T00:56:16.6259408Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6259826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6260174Z res = mod(**inputs) 2025-10-10T00:56:16.6260552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6260960Z outputs = self.fnet( 2025-10-10T00:56:16.6261344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-10-10T00:56:16.6261774Z embedding_output = self.embeddings( 2025-10-10T00:56:16.6262224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-10-10T00:56:16.6262697Z embeddings = self.projection(embeddings) 2025-10-10T00:56:16.6262854Z 2025-10-10T00:56:16.6262945Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6263216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6263609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6263961Z res = mod(**inputs) 2025-10-10T00:56:16.6264340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6264763Z outputs = self.fnet( 2025-10-10T00:56:16.6265150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6265578Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6265996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6266512Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6266945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6267355Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6267786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6268231Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6268684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6269123Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6269543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6269993Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6270166Z 2025-10-10T00:56:16.6270292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6270675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6271008Z res = mod(**inputs) 2025-10-10T00:56:16.6271377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6271783Z outputs = self.fnet( 2025-10-10T00:56:16.6272159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6272574Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6272985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6273426Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6273824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6274221Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6274663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6275078Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6275505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6275917Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6276330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6276769Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6276926Z 2025-10-10T00:56:16.6277038Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6277456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6277775Z res = mod(**inputs) 2025-10-10T00:56:16.6278134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6278520Z outputs = self.fnet( 2025-10-10T00:56:16.6278876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6279288Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6279693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6280122Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6280521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6280936Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6281322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6281738Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6282148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6282583Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6283002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6283417Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6283584Z 2025-10-10T00:56:16.6283692Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6284056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6284383Z res = mod(**inputs) 2025-10-10T00:56:16.6284742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6285130Z outputs = self.fnet( 2025-10-10T00:56:16.6285511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6285935Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6286339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6286769Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6287162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6287540Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6287956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6288398Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6288825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6289265Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6289681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6290119Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6290286Z 2025-10-10T00:56:16.6290399Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6290786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6291130Z res = mod(**inputs) 2025-10-10T00:56:16.6291511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6291983Z outputs = self.fnet( 2025-10-10T00:56:16.6292360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6292779Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6293190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6293622Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6294026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6294418Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6294838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6295269Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6295722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6296153Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6296605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6297109Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6297577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6298010Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6298166Z 2025-10-10T00:56:16.6298284Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6298679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6299034Z res = mod(**inputs) 2025-10-10T00:56:16.6299423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6300018Z outputs = self.fnet( 2025-10-10T00:56:16.6300397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6300810Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6301219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6301660Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6302051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6302455Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6302872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6303300Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6303762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6304200Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6304695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6305210Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6305682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6306166Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6306651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6307185Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6307469Z 2025-10-10T00:56:16.6307630Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6308042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6308397Z res = mod(**inputs) 2025-10-10T00:56:16.6308784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6309195Z outputs = self.fnet( 2025-10-10T00:56:16.6309583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6309993Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6310395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6310828Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6311222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6311596Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6311987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6312386Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6312797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6313203Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6313616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6314089Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6314527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6314961Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6315118Z 2025-10-10T00:56:16.6315209Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6315471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6315856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6316205Z res = mod(**inputs) 2025-10-10T00:56:16.6316586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6316989Z outputs = self.fnet( 2025-10-10T00:56:16.6317373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6317757Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6318142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6318574Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6318975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6319384Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6319795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6320235Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6320668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6321085Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6321493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6321938Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6322113Z 2025-10-10T00:56:16.6322275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6322667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6323016Z res = mod(**inputs) 2025-10-10T00:56:16.6323393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6323806Z outputs = self.fnet( 2025-10-10T00:56:16.6324191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6324604Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6325002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6325430Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6325832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6326229Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6326640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6327074Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6327510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6327937Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6328364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6328811Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6328979Z 2025-10-10T00:56:16.6329092Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6329493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6329830Z res = mod(**inputs) 2025-10-10T00:56:16.6330188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6330579Z outputs = self.fnet( 2025-10-10T00:56:16.6330936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6331326Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6331898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6332303Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6332679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6333055Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6333459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6333882Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6334297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6334739Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6335137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6335560Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6335722Z 2025-10-10T00:56:16.6335840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6336214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6336544Z res = mod(**inputs) 2025-10-10T00:56:16.6336960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6337365Z outputs = self.fnet( 2025-10-10T00:56:16.6337725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6338109Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6338493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6338895Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6339268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6339657Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6340041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6340461Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6340878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6341275Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6341672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6342108Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6342283Z 2025-10-10T00:56:16.6342394Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6342781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6343129Z res = mod(**inputs) 2025-10-10T00:56:16.6343500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6343921Z outputs = self.fnet( 2025-10-10T00:56:16.6344305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6344719Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6345123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6345544Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6345938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6346328Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6346811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6347239Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6347677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6348130Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6348578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6349108Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6349552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6349978Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6350134Z 2025-10-10T00:56:16.6350246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6350635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6350980Z res = mod(**inputs) 2025-10-10T00:56:16.6351351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6352545Z outputs = self.fnet( 2025-10-10T00:56:16.6352965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6353381Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6353781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6354169Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6354538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6354899Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6355290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6355691Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6356113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6356506Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6356916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6357367Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6357779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6358193Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6358577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6359034Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6359259Z 2025-10-10T00:56:16.6359376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6359736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6360061Z res = mod(**inputs) 2025-10-10T00:56:16.6360416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6360794Z outputs = self.fnet( 2025-10-10T00:56:16.6361150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6361520Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6361895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6362287Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6362655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6363014Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6363399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6363813Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6364213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6364606Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6365014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6365491Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6365929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6366327Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6366476Z 2025-10-10T00:56:16.6366616Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6366851Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6367211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6367537Z res = mod(**inputs) 2025-10-10T00:56:16.6367895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6368273Z outputs = self.fnet( 2025-10-10T00:56:16.6368639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6369028Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6369408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6369810Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6370173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6370538Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6370923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6371335Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6371747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6372138Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6387654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6388310Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6388537Z 2025-10-10T00:56:16.6388661Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6389080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6389430Z res = mod(**inputs) 2025-10-10T00:56:16.6389825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6390219Z outputs = self.fnet( 2025-10-10T00:56:16.6390603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6390990Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6391374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6391765Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6392142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6392514Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6392910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6393349Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6393906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6394335Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6394761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6395214Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6395391Z 2025-10-10T00:56:16.6395520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6395914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6396272Z res = mod(**inputs) 2025-10-10T00:56:16.6396761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6397176Z outputs = self.fnet( 2025-10-10T00:56:16.6397555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6397971Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6398379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6398809Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6399209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6399598Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6400016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6400461Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6400892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6401317Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6401726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6402169Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6402344Z 2025-10-10T00:56:16.6402462Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6402854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6403197Z res = mod(**inputs) 2025-10-10T00:56:16.6403581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6403989Z outputs = self.fnet( 2025-10-10T00:56:16.6404376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6404789Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6405190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6405629Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6406027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6406429Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6406823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6407270Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6407710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6408137Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6408549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6409023Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6409205Z 2025-10-10T00:56:16.6409321Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6409716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6410077Z res = mod(**inputs) 2025-10-10T00:56:16.6410466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6410877Z outputs = self.fnet( 2025-10-10T00:56:16.6411269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6411747Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6412156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6412598Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6412991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6413385Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6413811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6414248Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6414684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6415115Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6415571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6416070Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6416535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6416961Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6417120Z 2025-10-10T00:56:16.6417235Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6417645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6418005Z res = mod(**inputs) 2025-10-10T00:56:16.6418396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6418808Z outputs = self.fnet( 2025-10-10T00:56:16.6419200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6419626Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6420036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6420478Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6420893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6421303Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6421744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6422192Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6422634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6423088Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6423547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6424077Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6424551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6425023Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6425459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6425982Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6426239Z 2025-10-10T00:56:16.6426371Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6426863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6427290Z res = mod(**inputs) 2025-10-10T00:56:16.6427695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6428119Z outputs = self.fnet( 2025-10-10T00:56:16.6428523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6428955Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6429380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6429831Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6430243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6430653Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6431084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6431724Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6432202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6432654Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6433127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6433650Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6434143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6434594Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6434751Z 2025-10-10T00:56:16.6434857Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6435132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6435543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6435905Z res = mod(**inputs) 2025-10-10T00:56:16.6436310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6436734Z outputs = self.fnet( 2025-10-10T00:56:16.6437099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6437499Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6437892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6438312Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6438697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6439076Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6439477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6439977Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6440395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6440791Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6441192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6441631Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6441806Z 2025-10-10T00:56:16.6441932Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6442328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6442756Z res = mod(**inputs) 2025-10-10T00:56:16.6443140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6443556Z outputs = self.fnet( 2025-10-10T00:56:16.6443917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6444306Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6444685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6445090Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6445464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6445837Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6446227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6446650Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6447067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6447469Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6447862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6448272Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6448442Z 2025-10-10T00:56:16.6448552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6448921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6449256Z res = mod(**inputs) 2025-10-10T00:56:16.6449620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6450002Z outputs = self.fnet( 2025-10-10T00:56:16.6450366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6450762Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6451149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6451562Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6451961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6452357Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6452769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6453208Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6453627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6454022Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6454431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6454853Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6455013Z 2025-10-10T00:56:16.6455132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6455512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6455860Z res = mod(**inputs) 2025-10-10T00:56:16.6456246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6456654Z outputs = self.fnet( 2025-10-10T00:56:16.6457060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6457488Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6457897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6458325Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6458722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6459107Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6459523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6459959Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6460389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6460826Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6461241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6461675Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6461850Z 2025-10-10T00:56:16.6461964Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6462345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6462690Z res = mod(**inputs) 2025-10-10T00:56:16.6463067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6463461Z outputs = self.fnet( 2025-10-10T00:56:16.6463841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6464246Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6464653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6465071Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6465466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6465854Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6466271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6466792Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6467238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6467701Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6468150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6468643Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6469096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6469542Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6469704Z 2025-10-10T00:56:16.6469819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6470215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6470568Z res = mod(**inputs) 2025-10-10T00:56:16.6470947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6471357Z outputs = self.fnet( 2025-10-10T00:56:16.6471741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6472221Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6472642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6473080Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6473500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6473892Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6474301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6474736Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6475161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6475601Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6476050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6476548Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6477002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6477468Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6477880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6478372Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6478617Z 2025-10-10T00:56:16.6478740Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6479138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6479499Z res = mod(**inputs) 2025-10-10T00:56:16.6479907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6480326Z outputs = self.fnet( 2025-10-10T00:56:16.6480749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6481134Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6481545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6481981Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6482375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6482804Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6483223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6483650Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6484085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6484543Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6484980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6485490Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6485957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6486391Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6486540Z 2025-10-10T00:56:16.6486637Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6486891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6487315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6487677Z res = mod(**inputs) 2025-10-10T00:56:16.6488060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6488470Z outputs = self.fnet( 2025-10-10T00:56:16.6488846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6489264Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6489677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6490112Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6490500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6490899Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6491318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6491764Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6492214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6492651Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6493071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6493529Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6493697Z 2025-10-10T00:56:16.6493819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6494206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6494550Z res = mod(**inputs) 2025-10-10T00:56:16.6494935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6495347Z outputs = self.fnet( 2025-10-10T00:56:16.6495727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6496135Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6496542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6496964Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6497359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6497753Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6498160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6498604Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6499041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6499477Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6499893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6500332Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6500507Z 2025-10-10T00:56:16.6500619Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6501007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6501355Z res = mod(**inputs) 2025-10-10T00:56:16.6501725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6502126Z outputs = self.fnet( 2025-10-10T00:56:16.6502555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6502968Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6503374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6503791Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6504185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6504576Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6504986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6505425Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6505853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6506275Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6506776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6507246Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6507420Z 2025-10-10T00:56:16.6507538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6507947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6508296Z res = mod(**inputs) 2025-10-10T00:56:16.6508681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6509090Z outputs = self.fnet( 2025-10-10T00:56:16.6509477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6509908Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6510322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6510761Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6511158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6511565Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6511988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6512442Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6512884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6513318Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6513744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6514197Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6514366Z 2025-10-10T00:56:16.6514513Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6514909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6515260Z res = mod(**inputs) 2025-10-10T00:56:16.6515654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6516083Z outputs = self.fnet( 2025-10-10T00:56:16.6516473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6516904Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6517395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6517854Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6518264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6518675Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6519111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6519558Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6520001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6520463Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6520929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6521445Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6521927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6522398Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6522560Z 2025-10-10T00:56:16.6522677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6523093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6523454Z res = mod(**inputs) 2025-10-10T00:56:16.6523829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6524245Z outputs = self.fnet( 2025-10-10T00:56:16.6524641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6525079Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6525510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6525955Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6526364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6526775Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6527208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6527657Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6528121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6528562Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6529002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6529494Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6529941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6530408Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6530818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6531312Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6531733Z 2025-10-10T00:56:16.6531877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6532356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6532721Z res = mod(**inputs) 2025-10-10T00:56:16.6533200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6533635Z outputs = self.fnet( 2025-10-10T00:56:16.6534022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6534428Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6534834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6535267Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6535664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6536065Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6536476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6536900Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6537342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6537766Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6538203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6538704Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6539164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6539578Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6539717Z 2025-10-10T00:56:16.6539990Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6540225Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6540582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6540906Z res = mod(**inputs) 2025-10-10T00:56:16.6541263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6541637Z outputs = self.fnet( 2025-10-10T00:56:16.6541993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6542408Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6542813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6543239Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6543628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6544024Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6544441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6544879Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6545312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6545752Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6546166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6546674Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6546850Z 2025-10-10T00:56:16.6546975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6547373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6547722Z res = mod(**inputs) 2025-10-10T00:56:16.6548087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6548523Z outputs = self.fnet( 2025-10-10T00:56:16.6548877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6549259Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6549630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6550029Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6550401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6550763Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6551138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6551552Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6551964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6552384Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6552770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6553178Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6553342Z 2025-10-10T00:56:16.6553448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6553808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6554135Z res = mod(**inputs) 2025-10-10T00:56:16.6554493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6554864Z outputs = self.fnet( 2025-10-10T00:56:16.6555222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6555609Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6555997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6556399Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6556777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6557153Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6557558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6557967Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6558368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6558814Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6559220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6559644Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6559826Z 2025-10-10T00:56:16.6559939Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6560298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6560627Z res = mod(**inputs) 2025-10-10T00:56:16.6560986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6561370Z outputs = self.fnet( 2025-10-10T00:56:16.6561725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6562169Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6562583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6563012Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6563395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6563774Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6564177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6564602Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6565023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6565430Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6565823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6566258Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6566428Z 2025-10-10T00:56:16.6566539Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6566913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6567246Z res = mod(**inputs) 2025-10-10T00:56:16.6567618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6568015Z outputs = self.fnet( 2025-10-10T00:56:16.6568383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6568786Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6569171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6569583Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6569970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6570352Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6570749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6571162Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6571584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6571993Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6572424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6572892Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6573323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6573725Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6573866Z 2025-10-10T00:56:16.6573983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6574355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6574671Z res = mod(**inputs) 2025-10-10T00:56:16.6575024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6575409Z outputs = self.fnet( 2025-10-10T00:56:16.6575770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6576161Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6576542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6576984Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6577357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6577727Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6578105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6578501Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6578910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6579312Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6579723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6580178Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6580620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6581058Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6581472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6581978Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6582228Z 2025-10-10T00:56:16.6582347Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6582753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6583117Z res = mod(**inputs) 2025-10-10T00:56:16.6583508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6583921Z outputs = self.fnet( 2025-10-10T00:56:16.6584324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6584741Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6585165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6585603Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6586000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6586405Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6586911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6587347Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6587804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6588210Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6588622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6589112Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6589574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6590015Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6590165Z 2025-10-10T00:56:16.6590258Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6590525Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6590912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6591263Z res = mod(**inputs) 2025-10-10T00:56:16.6591666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6592094Z outputs = self.fnet( 2025-10-10T00:56:16.6592480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6592897Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6593307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6593760Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6594168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6594572Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6594993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6595450Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6595896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6596329Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6596746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6597203Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6597370Z 2025-10-10T00:56:16.6597484Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6597872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6598233Z res = mod(**inputs) 2025-10-10T00:56:16.6598627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6599056Z outputs = self.fnet( 2025-10-10T00:56:16.6599449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6599880Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6600296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6600730Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6601129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6601534Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6601946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6602382Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6602824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6603247Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6603672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6604148Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6604319Z 2025-10-10T00:56:16.6604442Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6604842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6605204Z res = mod(**inputs) 2025-10-10T00:56:16.6605589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6605997Z outputs = self.fnet( 2025-10-10T00:56:16.6606381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6606804Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6607266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6607709Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6607958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6608056Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6608325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6608444Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6608713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6608812Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6609088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6609204Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6609209Z 2025-10-10T00:56:16.6609336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6609554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6609635Z res = mod(**inputs) 2025-10-10T00:56:16.6609905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6609983Z outputs = self.fnet( 2025-10-10T00:56:16.6610262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6610344Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6610619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6610719Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6610968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6611059Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6611327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6611446Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6611715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6611813Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6612081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6612193Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6612199Z 2025-10-10T00:56:16.6612325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6612544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6612644Z res = mod(**inputs) 2025-10-10T00:56:16.6612915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6612995Z outputs = self.fnet( 2025-10-10T00:56:16.6613267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6613348Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6613626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6613720Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6614001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6614107Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6614379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6614484Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6614779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6614876Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6615184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6615311Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6615589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6615690Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6615694Z 2025-10-10T00:56:16.6615817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6616038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6616119Z res = mod(**inputs) 2025-10-10T00:56:16.6616392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6616469Z outputs = self.fnet( 2025-10-10T00:56:16.6616749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6616828Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6617108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6617201Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6617452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6617549Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6617823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6617924Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6618212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6618307Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6618621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6618749Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6619031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6619156Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6619401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6619626Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6619630Z 2025-10-10T00:56:16.6619746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6619970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6620044Z res = mod(**inputs) 2025-10-10T00:56:16.6620325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6620400Z outputs = self.fnet( 2025-10-10T00:56:16.6620707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6620814Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6621082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6621190Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6621435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6621532Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6621804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6621897Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6622191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6622277Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6622594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6622736Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6623018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6623111Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6623114Z 2025-10-10T00:56:16.6623206Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6623330Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6623549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6623629Z res = mod(**inputs) 2025-10-10T00:56:16.6623899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6623979Z outputs = self.fnet( 2025-10-10T00:56:16.6624259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6624344Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6624621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6624715Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6624960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6625056Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6625325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6625441Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6625714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6625813Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6626083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6626217Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6626221Z 2025-10-10T00:56:16.6626345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6626635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6626722Z res = mod(**inputs) 2025-10-10T00:56:16.6626992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6627069Z outputs = self.fnet( 2025-10-10T00:56:16.6627390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6627489Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6627772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6627873Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6628133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6628223Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6628496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6628619Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6628892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6628990Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6629266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6629381Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6629387Z 2025-10-10T00:56:16.6629508Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6629727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6629805Z res = mod(**inputs) 2025-10-10T00:56:16.6630078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6630153Z outputs = self.fnet( 2025-10-10T00:56:16.6630431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6630520Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6630795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6630888Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6631131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6631219Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6631595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6631730Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6631991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6632090Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6632352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6632471Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6632485Z 2025-10-10T00:56:16.6632596Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6632807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6632938Z res = mod(**inputs) 2025-10-10T00:56:16.6633204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6633283Z outputs = self.fnet( 2025-10-10T00:56:16.6633549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6633627Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6633898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6633989Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6634314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6634403Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6634668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6634779Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6635044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6635138Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6635403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6635517Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6635520Z 2025-10-10T00:56:16.6635631Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6635845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6635924Z res = mod(**inputs) 2025-10-10T00:56:16.6636190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6636271Z outputs = self.fnet( 2025-10-10T00:56:16.6636535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6636612Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6636886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6636978Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6637225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6637317Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6637583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6637684Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6637965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6638058Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6638356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6638487Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6638750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6638841Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6638846Z 2025-10-10T00:56:16.6638969Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6639182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6639284Z res = mod(**inputs) 2025-10-10T00:56:16.6639558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6639631Z outputs = self.fnet( 2025-10-10T00:56:16.6639911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6639989Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6640267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6640362Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6640645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6640747Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6641008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6641108Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6641385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6641482Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6641757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6641868Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6642119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6642229Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6642451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6642631Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6642636Z 2025-10-10T00:56:16.6642746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6642942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6643007Z res = mod(**inputs) 2025-10-10T00:56:16.6643259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6643326Z outputs = self.fnet( 2025-10-10T00:56:16.6643575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6643648Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6643894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6643986Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6644204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6644294Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6644536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6644626Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6644881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6644959Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6645238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6645368Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6645618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6645716Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6645720Z 2025-10-10T00:56:16.6645803Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6645917Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6646118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6646190Z res = mod(**inputs) 2025-10-10T00:56:16.6646444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6646514Z outputs = self.fnet( 2025-10-10T00:56:16.6646811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6646903Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6647159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6647246Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6647480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6647561Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6647857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6647963Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6648204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6648296Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6648537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6648642Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6648653Z 2025-10-10T00:56:16.6648757Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6648952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6649022Z res = mod(**inputs) 2025-10-10T00:56:16.6649269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6649344Z outputs = self.fnet( 2025-10-10T00:56:16.6649593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6649666Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6649931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6650015Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6650241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6650320Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6650561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6650665Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6650907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6650992Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6651238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6651342Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6651353Z 2025-10-10T00:56:16.6651453Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6651664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6651737Z res = mod(**inputs) 2025-10-10T00:56:16.6651978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6652052Z outputs = self.fnet( 2025-10-10T00:56:16.6652294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6652366Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6652612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6652740Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6652963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6653044Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6653285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6653387Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6653630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6653716Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6653958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6654067Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6654073Z 2025-10-10T00:56:16.6654177Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6654373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6654447Z res = mod(**inputs) 2025-10-10T00:56:16.6654690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6654765Z outputs = self.fnet( 2025-10-10T00:56:16.6655008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6655081Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6655328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6655415Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6655641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6655722Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6655971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6656071Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6656313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6656402Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6656645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6656753Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6656756Z 2025-10-10T00:56:16.6656858Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6657056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6657131Z res = mod(**inputs) 2025-10-10T00:56:16.6657370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6657463Z outputs = self.fnet( 2025-10-10T00:56:16.6657702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6657774Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6658019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6658103Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6658324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6658403Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6658684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6658784Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6659059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6659145Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6659414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6659534Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6659774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6659858Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6659868Z 2025-10-10T00:56:16.6659971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6660169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6660242Z res = mod(**inputs) 2025-10-10T00:56:16.6660484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6660563Z outputs = self.fnet( 2025-10-10T00:56:16.6660805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6660878Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6661135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6661220Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6661448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6661530Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6661780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6661872Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6662142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6662232Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6662527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6662659Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6662922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6663040Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6663277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6663469Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6663491Z 2025-10-10T00:56:16.6663613Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6663844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6663915Z res = mod(**inputs) 2025-10-10T00:56:16.6664185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6664258Z outputs = self.fnet( 2025-10-10T00:56:16.6664531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6664609Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6664917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6665026Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6665264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6665358Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6665619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6665712Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6665988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6666070Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6666379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6666600Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6666892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6666986Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6666991Z 2025-10-10T00:56:16.6667090Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6667206Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6667440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6667519Z res = mod(**inputs) 2025-10-10T00:56:16.6667794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6667877Z outputs = self.fnet( 2025-10-10T00:56:16.6668156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6668233Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6668477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6668562Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6668781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6668861Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6669100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6669205Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6669443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6669529Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6669770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6669880Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6669902Z 2025-10-10T00:56:16.6670003Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6670195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6670265Z res = mod(**inputs) 2025-10-10T00:56:16.6670504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6670577Z outputs = self.fnet( 2025-10-10T00:56:16.6670815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6670886Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6671197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6671296Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6671516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6671595Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6671839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6671945Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6672190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6672278Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6672524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6672637Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6672642Z 2025-10-10T00:56:16.6672743Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6672939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6673013Z res = mod(**inputs) 2025-10-10T00:56:16.6673260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6673334Z outputs = self.fnet( 2025-10-10T00:56:16.6673578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6673650Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6673907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6673990Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6674216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6674294Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6674541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6674639Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6674877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6674965Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6675203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6675309Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6675312Z 2025-10-10T00:56:16.6675411Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6675604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6675675Z res = mod(**inputs) 2025-10-10T00:56:16.6675911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6676001Z outputs = self.fnet( 2025-10-10T00:56:16.6676235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6676314Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6676548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6676632Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6676853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6676930Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6677217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6677313Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6677547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6677632Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6677866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6677972Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6677975Z 2025-10-10T00:56:16.6678073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6678271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6678332Z res = mod(**inputs) 2025-10-10T00:56:16.6678573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6678643Z outputs = self.fnet( 2025-10-10T00:56:16.6678880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6678957Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6679191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6679274Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6679494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6679571Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6679817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6679904Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6680162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6680247Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6680521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6680642Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6680882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6680970Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6680974Z 2025-10-10T00:56:16.6681073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6681268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6681343Z res = mod(**inputs) 2025-10-10T00:56:16.6681593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6681682Z outputs = self.fnet( 2025-10-10T00:56:16.6681917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6681988Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6682234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6682319Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6682542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6682621Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6682891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6682997Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6683255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6683347Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6683643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6683771Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6684036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6684147Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6684373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6684559Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6684563Z 2025-10-10T00:56:16.6684675Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6684875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6684949Z res = mod(**inputs) 2025-10-10T00:56:16.6685208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6685277Z outputs = self.fnet( 2025-10-10T00:56:16.6685524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6685598Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6685879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6685967Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6686185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6686275Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6686516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6686607Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6686860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6686938Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6687216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6687344Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6687597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6687679Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6687700Z 2025-10-10T00:56:16.6687791Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6687892Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6688090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6688161Z res = mod(**inputs) 2025-10-10T00:56:16.6688404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6688477Z outputs = self.fnet( 2025-10-10T00:56:16.6688720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6688793Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6689089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6689175Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6689402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6689480Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6689727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6689825Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6690067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6690155Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6690399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6690514Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6690517Z 2025-10-10T00:56:16.6690618Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6690816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6690889Z res = mod(**inputs) 2025-10-10T00:56:16.6691132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6691207Z outputs = self.fnet( 2025-10-10T00:56:16.6691448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6691519Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6691766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6691852Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6692078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6692158Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6692410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6692509Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6692753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6692839Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6693083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6693192Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6693196Z 2025-10-10T00:56:16.6693301Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6693495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6693567Z res = mod(**inputs) 2025-10-10T00:56:16.6693839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6693914Z outputs = self.fnet( 2025-10-10T00:56:16.6694210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6694288Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6694528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6694612Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6694836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6694964Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6695222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6695321Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6695571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6695659Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6695907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6696016Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6696020Z 2025-10-10T00:56:16.6696133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6696334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6696400Z res = mod(**inputs) 2025-10-10T00:56:16.6696642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6696718Z outputs = self.fnet( 2025-10-10T00:56:16.6696959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6697037Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6697281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6697365Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6697593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6697671Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6697921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6698019Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6698260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6698347Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6698589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6698697Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6698700Z 2025-10-10T00:56:16.6698799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6699002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6699069Z res = mod(**inputs) 2025-10-10T00:56:16.6699319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6699396Z outputs = self.fnet( 2025-10-10T00:56:16.6699644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6699742Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6699992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6700080Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6700314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6700395Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6700654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6700741Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6701059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6701139Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6701420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6701545Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6701791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6701885Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6701889Z 2025-10-10T00:56:16.6701995Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6702210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6702284Z res = mod(**inputs) 2025-10-10T00:56:16.6702551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6702630Z outputs = self.fnet( 2025-10-10T00:56:16.6702891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6702979Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6703241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6703333Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6703581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6703666Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6703934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6704028Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6704306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6704399Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6704695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6704825Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6705089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6705208Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6705445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6705639Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6705644Z 2025-10-10T00:56:16.6705762Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6705984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6706080Z res = mod(**inputs) 2025-10-10T00:56:16.6706362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6706504Z outputs = self.fnet( 2025-10-10T00:56:16.6706807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6706891Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6707178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6707273Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6707575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6707674Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6707951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6708057Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6708343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6708436Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6708751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6708893Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6709191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6709283Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6709287Z 2025-10-10T00:56:16.6709384Z cudagraph partition due to non gpu ops 2025-10-10T00:56:16.6709497Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6709724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6709802Z res = mod(**inputs) 2025-10-10T00:56:16.6710076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6710157Z outputs = self.fnet( 2025-10-10T00:56:16.6710430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6710516Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6710790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6710883Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6711127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6711213Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6711502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6711610Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6711899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6711996Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6712279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6712402Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6712411Z 2025-10-10T00:56:16.6712524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6712753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6712848Z res = mod(**inputs) 2025-10-10T00:56:16.6713141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6713226Z outputs = self.fnet( 2025-10-10T00:56:16.6713520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6713606Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6713891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6713986Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6714272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6714389Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6714678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6714785Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6715056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6715151Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6715422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6715542Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6715545Z 2025-10-10T00:56:16.6715655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6715875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6715947Z res = mod(**inputs) 2025-10-10T00:56:16.6716207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6716288Z outputs = self.fnet( 2025-10-10T00:56:16.6716551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6716636Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6716899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6716989Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6717235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6717321Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6717594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6717698Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6717978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6718067Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6718337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6718466Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6718470Z 2025-10-10T00:56:16.6718579Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6718794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6718863Z res = mod(**inputs) 2025-10-10T00:56:16.6719137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6719231Z outputs = self.fnet( 2025-10-10T00:56:16.6719503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6719611Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6719883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6719976Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6720228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6720315Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6720599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-10-10T00:56:16.6720760Z self_fourier_outputs = self.fourier(hidden_states) 2025-10-10T00:56:16.6721037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-10-10T00:56:16.6721128Z self_outputs = self.self(hidden_states) 2025-10-10T00:56:16.6721398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-10-10T00:56:16.6721519Z outputs = self.fourier_transform(hidden_states).real 2025-10-10T00:56:16.6721523Z 2025-10-10T00:56:16.6721635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6721858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6721930Z res = mod(**inputs) 2025-10-10T00:56:16.6722199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6722280Z outputs = self.fnet( 2025-10-10T00:56:16.6722554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6722643Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6722914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6723014Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6723257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6723346Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6723624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6723718Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6724015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6724102Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6724408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6724544Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6724815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-10-10T00:56:16.6724912Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6724916Z 2025-10-10T00:56:16.6725028Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6725255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6725338Z res = mod(**inputs) 2025-10-10T00:56:16.6725608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6725696Z outputs = self.fnet( 2025-10-10T00:56:16.6725966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6726072Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6726338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6726431Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6726684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6726771Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6727046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6727137Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6727453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6727562Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6727866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-10-10T00:56:16.6728002Z intermediate_output = self.intermediate(fourier_output) 2025-10-10T00:56:16.6728270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-10-10T00:56:16.6728396Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:16.6728630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-10-10T00:56:16.6728826Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-10-10T00:56:16.6728830Z 2025-10-10T00:56:16.6728956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6729175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6729254Z res = mod(**inputs) 2025-10-10T00:56:16.6729527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-10-10T00:56:16.6729609Z outputs = self.fnet( 2025-10-10T00:56:16.6729880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-10-10T00:56:16.6729961Z encoder_outputs = self.encoder( 2025-10-10T00:56:16.6730238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-10-10T00:56:16.6730331Z layer_outputs = layer_module(hidden_states) 2025-10-10T00:56:16.6730579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:16.6730671Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:16.6730942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-10-10T00:56:16.6731044Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:16.6731330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:16.6731422Z return forward_fn(*input_tensors) 2025-10-10T00:56:16.6731894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-10-10T00:56:16.6732046Z layer_output = self.output(intermediate_output, fourier_output) 2025-10-10T00:56:16.6732330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-10-10T00:56:16.6732422Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6732432Z 2025-10-10T00:56:16.6732554Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6732773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6732898Z res = mod(**inputs) 2025-10-10T00:56:16.6733170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-10-10T00:56:16.6733285Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:56:16.6733561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-10-10T00:56:16.6733685Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:56:16.6733957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-10-10T00:56:16.6734054Z hidden_states = self.transform(hidden_states) 2025-10-10T00:56:16.6734373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-10-10T00:56:16.6734470Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:16.6734475Z 2025-10-10T00:56:16.6734580Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6734792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6734860Z res = mod(**inputs) 2025-10-10T00:56:16.6735119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-10-10T00:56:16.6735220Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:56:16.6735482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-10-10T00:56:16.6735608Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:56:16.6735874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-10-10T00:56:16.6735974Z hidden_states = self.decoder(hidden_states) 2025-10-10T00:56:16.6735979Z 2025-10-10T00:56:16.6736087Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:16.6736298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:16.6736374Z res = mod(**inputs) 2025-10-10T00:56:16.6736635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-10-10T00:56:16.6736843Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:56:16.6736847Z 2025-10-10T00:56:26.7574791Z Compilation time (from dynamo_timed): 14.655716207 2025-10-10T00:56:26.7612470Z pass 2025-10-10T00:56:26.7616392Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:26.7617413Z TIMING: _recursive_pre_grad_passes:0.00627 _recursive_joint_graph_passes:0.22674 _recursive_post_grad_passes:0.06934 async_compile.wait:0.7052 code_gen:9.6846 inductor_compile:10.92623 backend_compile:12.83502 gc:0.00145 entire_frame_compile:14.65572 total_wall_time:14.65572 2025-10-10T00:56:26.7618420Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:4285 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:1233 2025-10-10T00:56:26.7620846Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-10-10T00:56:29.4944298Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:56:29.4945214Z import pynvml # type: ignore[import] 2025-10-10T00:56:33.0570266Z 2025-10-10T00:56:34.3065604Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:56:34.3065923Z loading model: 0it [00:01, ?it/s] 2025-10-10T00:56:34.3087433Z cpu eval LayoutLMForMaskedLM 2025-10-10T00:56:34.9497650Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:35.2132918Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:35.5215377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:44.1894439Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1894980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1895362Z res = mod(**inputs) 2025-10-10T00:56:44.1895770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1896692Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1897187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1897664Z outputs = self.layoutlm( 2025-10-10T00:56:44.1898062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1898471Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1898962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1899419Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1899819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1900208Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1900731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1901192Z layer_outputs = layer_module( 2025-10-10T00:56:44.1901595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1902015Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1902474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1902948Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1903414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1903864Z self_outputs = self.self( 2025-10-10T00:56:44.1904296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.1904825Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1905069Z 2025-10-10T00:56:44.1905193Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1905612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1905980Z res = mod(**inputs) 2025-10-10T00:56:44.1906544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1906943Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1907398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1907847Z outputs = self.layoutlm( 2025-10-10T00:56:44.1908239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1908624Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1909061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1909503Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1909968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1910359Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1910803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1911252Z layer_outputs = layer_module( 2025-10-10T00:56:44.1911638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1912049Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1912489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1912999Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1913475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1913928Z self_outputs = self.self( 2025-10-10T00:56:44.1914364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.1914886Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1915107Z 2025-10-10T00:56:44.1915228Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1915638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1916049Z res = mod(**inputs) 2025-10-10T00:56:44.1916418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1917071Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1917516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1917964Z outputs = self.layoutlm( 2025-10-10T00:56:44.1918347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1918748Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1919225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1919680Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1920072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1920477Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1920932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1921388Z layer_outputs = layer_module( 2025-10-10T00:56:44.1921774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1922195Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1922649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1923108Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1923569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1924018Z self_outputs = self.self( 2025-10-10T00:56:44.1924455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.1924992Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1925222Z 2025-10-10T00:56:44.1925325Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.1925573Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.1925862Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1926262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1926624Z res = mod(**inputs) 2025-10-10T00:56:44.1926998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1927382Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1927828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1928271Z outputs = self.layoutlm( 2025-10-10T00:56:44.1928646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1929109Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1929547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1929999Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1930389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1930770Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1931203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1931937Z layer_outputs = layer_module( 2025-10-10T00:56:44.1932343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1932756Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1933201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1933641Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1934090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.1934586Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.1935081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.1935532Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.1935686Z 2025-10-10T00:56:44.1935804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1936195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1936545Z res = mod(**inputs) 2025-10-10T00:56:44.1936901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1937281Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1937708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1938138Z outputs = self.layoutlm( 2025-10-10T00:56:44.1938502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1938879Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1939307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1939739Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1940114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1940495Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1940928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1941353Z layer_outputs = layer_module( 2025-10-10T00:56:44.1941790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1942182Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1942617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.1943060Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.1943490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.1943918Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.1944482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.1945038Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.1945525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.1945968Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.1946126Z 2025-10-10T00:56:44.1946242Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1946709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1947067Z res = mod(**inputs) 2025-10-10T00:56:44.1947425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1947830Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1948270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1948700Z outputs = self.layoutlm( 2025-10-10T00:56:44.1949065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1949438Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1949840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1950237Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1950589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1950942Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1951349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1951757Z layer_outputs = layer_module( 2025-10-10T00:56:44.1952125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1952517Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1952948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.1953394Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.1953832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.1954251Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.1954676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.1955140Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.1955591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.1956039Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.1956429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.1956806Z return self.act(input) 2025-10-10T00:56:44.1956924Z 2025-10-10T00:56:44.1957033Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1957404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1957730Z res = mod(**inputs) 2025-10-10T00:56:44.1958063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1958417Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1958825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1959291Z outputs = self.layoutlm( 2025-10-10T00:56:44.1959639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1960003Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1960410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1960823Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1961180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1961541Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1961944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1962355Z layer_outputs = layer_module( 2025-10-10T00:56:44.1962718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1963095Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1963504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.1963922Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.1964336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.1964747Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.1965199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.1965704Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.1966171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.1966600Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.1966750Z 2025-10-10T00:56:44.1966860Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1967269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1967645Z res = mod(**inputs) 2025-10-10T00:56:44.1967976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1968344Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1968758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1969168Z outputs = self.layoutlm( 2025-10-10T00:56:44.1969510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1969877Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1970295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1970711Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1971091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1971470Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1971880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1972285Z layer_outputs = layer_module( 2025-10-10T00:56:44.1972647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1973046Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1973468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1973952Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1974396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1974821Z self_outputs = self.self( 2025-10-10T00:56:44.1975218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.1975710Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1975925Z 2025-10-10T00:56:44.1976036Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1976413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1976747Z res = mod(**inputs) 2025-10-10T00:56:44.1977080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1977458Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1977876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1978287Z outputs = self.layoutlm( 2025-10-10T00:56:44.1978635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1978999Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1979412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1979826Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1980187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1980549Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1980969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1981399Z layer_outputs = layer_module( 2025-10-10T00:56:44.1981781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1982169Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1982580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1983004Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1983448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1983885Z self_outputs = self.self( 2025-10-10T00:56:44.1984308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.1984808Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1985029Z 2025-10-10T00:56:44.1985146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1985545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1985920Z res = mod(**inputs) 2025-10-10T00:56:44.1986351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1986762Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1987200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1987632Z outputs = self.layoutlm( 2025-10-10T00:56:44.1987999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1988386Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1988886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1989323Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1989706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1990098Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1990531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.1990965Z layer_outputs = layer_module( 2025-10-10T00:56:44.1991341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.1991737Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.1992164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.1992617Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.1993058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.1993490Z self_outputs = self.self( 2025-10-10T00:56:44.1993907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.1994389Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.1994597Z 2025-10-10T00:56:44.1994680Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.1994904Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.1995151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.1995530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.1995845Z res = mod(**inputs) 2025-10-10T00:56:44.1996173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1996529Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1996925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.1997318Z outputs = self.layoutlm( 2025-10-10T00:56:44.1997652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1998001Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1998397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.1998795Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.1999140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.1999507Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.1999910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2000357Z layer_outputs = layer_module( 2025-10-10T00:56:44.2000707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2001076Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2001484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2001899Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2002312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2002771Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2003302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2003754Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2003899Z 2025-10-10T00:56:44.2004017Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2004387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2004713Z res = mod(**inputs) 2025-10-10T00:56:44.2005047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2005413Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2005809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2006196Z outputs = self.layoutlm( 2025-10-10T00:56:44.2006535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2006885Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2007284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2007687Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2008025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2008374Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2008766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2009161Z layer_outputs = layer_module( 2025-10-10T00:56:44.2009502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2009856Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2010260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2010673Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2011076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2011460Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2011890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2012365Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2012808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2013219Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2013360Z 2025-10-10T00:56:44.2013469Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2013843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2014175Z res = mod(**inputs) 2025-10-10T00:56:44.2014537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2014901Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2015284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2015670Z outputs = self.layoutlm( 2025-10-10T00:56:44.2015995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2016363Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2016760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2017207Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2017545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2017890Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2018286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2018674Z layer_outputs = layer_module( 2025-10-10T00:56:44.2019022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2019383Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2019797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2020222Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2020631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2021038Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2021481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2021972Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2022427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2022872Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2023287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2023656Z return self.act(input) 2025-10-10T00:56:44.2023780Z 2025-10-10T00:56:44.2023901Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2024286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2024637Z res = mod(**inputs) 2025-10-10T00:56:44.2024988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2025375Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2025809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2026233Z outputs = self.layoutlm( 2025-10-10T00:56:44.2026686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2027069Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2027513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2027917Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2028280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2028642Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2029108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2029537Z layer_outputs = layer_module( 2025-10-10T00:56:44.2029880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2030252Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2030667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2031090Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2031624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2032142Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2032582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2033088Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2033559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2033981Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2034124Z 2025-10-10T00:56:44.2034232Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2034604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2034938Z res = mod(**inputs) 2025-10-10T00:56:44.2035271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2035636Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2036049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2036459Z outputs = self.layoutlm( 2025-10-10T00:56:44.2036806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2037166Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2037569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2037979Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2038335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2038696Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2039104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2039507Z layer_outputs = layer_module( 2025-10-10T00:56:44.2039868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2040243Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2040655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2041073Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2041504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2041902Z self_outputs = self.self( 2025-10-10T00:56:44.2042287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2042763Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2042966Z 2025-10-10T00:56:44.2043074Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2043492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2043842Z res = mod(**inputs) 2025-10-10T00:56:44.2044200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2044563Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2044966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2045383Z outputs = self.layoutlm( 2025-10-10T00:56:44.2045719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2046082Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2046543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2046932Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2047270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2047611Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2047996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2048375Z layer_outputs = layer_module( 2025-10-10T00:56:44.2048717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2049071Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2049465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2049865Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2050252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2050641Z self_outputs = self.self( 2025-10-10T00:56:44.2051020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2051465Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2051651Z 2025-10-10T00:56:44.2051753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2052107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2052428Z res = mod(**inputs) 2025-10-10T00:56:44.2052758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2053122Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2053524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2053941Z outputs = self.layoutlm( 2025-10-10T00:56:44.2054300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2054672Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2055070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2055462Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2055807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2056161Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2056563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2056950Z layer_outputs = layer_module( 2025-10-10T00:56:44.2057298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2057685Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2058076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2058473Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2058861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2059249Z self_outputs = self.self( 2025-10-10T00:56:44.2059630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2060173Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2060393Z 2025-10-10T00:56:44.2060484Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2060695Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2060952Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2061333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2061671Z res = mod(**inputs) 2025-10-10T00:56:44.2062004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2062380Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2062797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2063234Z outputs = self.layoutlm( 2025-10-10T00:56:44.2063611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2064005Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2064454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2064914Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2065298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2065686Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2066135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2066676Z layer_outputs = layer_module( 2025-10-10T00:56:44.2067074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2067496Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2067911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2068339Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2068742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2069194Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2069640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2070041Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2070187Z 2025-10-10T00:56:44.2070294Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2070654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2070983Z res = mod(**inputs) 2025-10-10T00:56:44.2071320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2071670Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2072097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2072491Z outputs = self.layoutlm( 2025-10-10T00:56:44.2072825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2073172Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2073581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2073991Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2074356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2074800Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2075214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2075626Z layer_outputs = layer_module( 2025-10-10T00:56:44.2075983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2076361Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2076757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2077168Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2077571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2077966Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2078402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2078884Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2079340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2079758Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2079899Z 2025-10-10T00:56:44.2080014Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2080381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2080705Z res = mod(**inputs) 2025-10-10T00:56:44.2081045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2081408Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2081819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2082227Z outputs = self.layoutlm( 2025-10-10T00:56:44.2082562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2082925Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2083352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2083790Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2084138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2084506Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2084936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2085358Z layer_outputs = layer_module( 2025-10-10T00:56:44.2085717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2086081Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2086521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2086941Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2087352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2087751Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2088192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2088681Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2089175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2089641Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2090023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2090375Z return self.act(input) 2025-10-10T00:56:44.2090497Z 2025-10-10T00:56:44.2090604Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2090976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2091308Z res = mod(**inputs) 2025-10-10T00:56:44.2091633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2091996Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2092415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2092851Z outputs = self.layoutlm( 2025-10-10T00:56:44.2093207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2093593Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2094025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2094446Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2094801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2095153Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2095564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2095971Z layer_outputs = layer_module( 2025-10-10T00:56:44.2096329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2096702Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2097108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2097531Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2097940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2098341Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2098769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2099265Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2099746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2100193Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2100342Z 2025-10-10T00:56:44.2100464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2100876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2101206Z res = mod(**inputs) 2025-10-10T00:56:44.2101534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2101893Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2102299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2102715Z outputs = self.layoutlm( 2025-10-10T00:56:44.2103084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2103469Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2103978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2104447Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2104825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2105210Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2105641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2106073Z layer_outputs = layer_module( 2025-10-10T00:56:44.2106536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2106988Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2107451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2107915Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2108382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2108815Z self_outputs = self.self( 2025-10-10T00:56:44.2109243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2109759Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2109979Z 2025-10-10T00:56:44.2110107Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2110519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2110886Z res = mod(**inputs) 2025-10-10T00:56:44.2111253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2111660Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2112113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2112554Z outputs = self.layoutlm( 2025-10-10T00:56:44.2112934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2113321Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2113759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2114193Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2114566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2114956Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2115394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2115834Z layer_outputs = layer_module( 2025-10-10T00:56:44.2116228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2116655Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2117112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2117554Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2117991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2118414Z self_outputs = self.self( 2025-10-10T00:56:44.2118828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2119419Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2119631Z 2025-10-10T00:56:44.2119753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2120147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2120487Z res = mod(**inputs) 2025-10-10T00:56:44.2120839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2121230Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2121663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2122099Z outputs = self.layoutlm( 2025-10-10T00:56:44.2122470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2122863Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2123319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2123766Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2124144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2124536Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2124982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2125428Z layer_outputs = layer_module( 2025-10-10T00:56:44.2125813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2126211Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2126661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2127123Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2127575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2128020Z self_outputs = self.self( 2025-10-10T00:56:44.2128446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2128974Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2129204Z 2025-10-10T00:56:44.2129295Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2129537Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2129799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2130202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2130564Z res = mod(**inputs) 2025-10-10T00:56:44.2130943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2131342Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2131944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2132396Z outputs = self.layoutlm( 2025-10-10T00:56:44.2132775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2133179Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2133621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2134070Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2134460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2134967Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2135421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2135869Z layer_outputs = layer_module( 2025-10-10T00:56:44.2136271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2136673Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2137114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2137558Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2137994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2138488Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2138983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2139432Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2139584Z 2025-10-10T00:56:44.2139708Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2140092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2140444Z res = mod(**inputs) 2025-10-10T00:56:44.2140800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2141183Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2141627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2142074Z outputs = self.layoutlm( 2025-10-10T00:56:44.2142449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2142844Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2143291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2143732Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2144121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2144510Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2144960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2145398Z layer_outputs = layer_module( 2025-10-10T00:56:44.2145797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2146203Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2146737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2147198Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2147698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2148128Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2148593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2149120Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2149602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2150034Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2150191Z 2025-10-10T00:56:44.2150362Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2150752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2151117Z res = mod(**inputs) 2025-10-10T00:56:44.2151455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2151812Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2152222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2152638Z outputs = self.layoutlm( 2025-10-10T00:56:44.2152975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2153317Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2153715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2154150Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2154529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2154909Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2155333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2155772Z layer_outputs = layer_module( 2025-10-10T00:56:44.2156128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2156501Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2156909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2157333Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2157764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2158191Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2158656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2159154Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2159609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2160066Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2160446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2160786Z return self.act(input) 2025-10-10T00:56:44.2160900Z 2025-10-10T00:56:44.2161006Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2161376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2161707Z res = mod(**inputs) 2025-10-10T00:56:44.2162043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2162417Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2162820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2163238Z outputs = self.layoutlm( 2025-10-10T00:56:44.2163602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2163981Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2164384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2164798Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2165210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2165819Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2166251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2166663Z layer_outputs = layer_module( 2025-10-10T00:56:44.2167037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2167414Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2167831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2168251Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2168675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2169090Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2169538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2170054Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2170525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2170959Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2171112Z 2025-10-10T00:56:44.2171221Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2171597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2171937Z res = mod(**inputs) 2025-10-10T00:56:44.2172276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2172669Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2173111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2173555Z outputs = self.layoutlm( 2025-10-10T00:56:44.2173919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2174280Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2174700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2175121Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2175483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2175845Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2176261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2176684Z layer_outputs = layer_module( 2025-10-10T00:56:44.2177047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2177444Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2177851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2178271Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2178691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2179097Z self_outputs = self.self( 2025-10-10T00:56:44.2179491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2180021Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2180237Z 2025-10-10T00:56:44.2180345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2180717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2181048Z res = mod(**inputs) 2025-10-10T00:56:44.2181373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2181764Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2182195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2182625Z outputs = self.layoutlm( 2025-10-10T00:56:44.2182985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2183365Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2183800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2184240Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2184614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2185002Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2185424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2185856Z layer_outputs = layer_module( 2025-10-10T00:56:44.2186243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2186738Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2187174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2187625Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2188050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2188452Z self_outputs = self.self( 2025-10-10T00:56:44.2188843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2189303Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2189505Z 2025-10-10T00:56:44.2189611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2189987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2190372Z res = mod(**inputs) 2025-10-10T00:56:44.2190731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2191124Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2191566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2191992Z outputs = self.layoutlm( 2025-10-10T00:56:44.2192356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2192736Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2193167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2193605Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2193958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2194319Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2194748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2195175Z layer_outputs = layer_module( 2025-10-10T00:56:44.2195531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2195912Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2196348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2196787Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2197222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2197651Z self_outputs = self.self( 2025-10-10T00:56:44.2198055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2198533Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2198742Z 2025-10-10T00:56:44.2198831Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2199067Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2199325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2199711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2200052Z res = mod(**inputs) 2025-10-10T00:56:44.2200402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2200784Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2201223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2201645Z outputs = self.layoutlm( 2025-10-10T00:56:44.2202010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2202393Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2202827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2203266Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2203634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2204020Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2204451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2204882Z layer_outputs = layer_module( 2025-10-10T00:56:44.2205259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2205644Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2206086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2206529Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2206991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2207473Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2207962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2208406Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2208557Z 2025-10-10T00:56:44.2208679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2209072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2209419Z res = mod(**inputs) 2025-10-10T00:56:44.2209831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2210211Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2210646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2211072Z outputs = self.layoutlm( 2025-10-10T00:56:44.2211441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2211838Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2212268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2212699Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2213066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2213449Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2213884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2214317Z layer_outputs = layer_module( 2025-10-10T00:56:44.2214707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2215115Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2215549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2215991Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2216425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2216846Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2217306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2217819Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2218299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2218737Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2218886Z 2025-10-10T00:56:44.2219009Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2219389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2219738Z res = mod(**inputs) 2025-10-10T00:56:44.2220094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2220484Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2220909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2221339Z outputs = self.layoutlm( 2025-10-10T00:56:44.2221701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2222120Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2222555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2222995Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2223375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2223777Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2224225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2224668Z layer_outputs = layer_module( 2025-10-10T00:56:44.2225096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2225509Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2225971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2226587Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2227039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2227488Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2227982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2228533Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2229045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2229545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2229982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2230374Z return self.act(input) 2025-10-10T00:56:44.2230500Z 2025-10-10T00:56:44.2230628Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2231037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2231406Z res = mod(**inputs) 2025-10-10T00:56:44.2231955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2232359Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2232817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2233261Z outputs = self.layoutlm( 2025-10-10T00:56:44.2233648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2234049Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2234498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2234950Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2235343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2235754Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2236210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2236641Z layer_outputs = layer_module( 2025-10-10T00:56:44.2237027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2237398Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2237810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2238275Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2238689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2239089Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2239530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2240033Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2240553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2241000Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2241143Z 2025-10-10T00:56:44.2241251Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2241625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2241960Z res = mod(**inputs) 2025-10-10T00:56:44.2242308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2242708Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2243136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2243569Z outputs = self.layoutlm( 2025-10-10T00:56:44.2243935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2244322Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2244748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2245188Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2245563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2245956Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2246372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2246773Z layer_outputs = layer_module( 2025-10-10T00:56:44.2247130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2247518Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2247957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2248401Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2248838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2249271Z self_outputs = self.self( 2025-10-10T00:56:44.2249692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2250203Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2250407Z 2025-10-10T00:56:44.2250515Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2250884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2251214Z res = mod(**inputs) 2025-10-10T00:56:44.2251552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2251920Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2252348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2252800Z outputs = self.layoutlm( 2025-10-10T00:56:44.2253160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2253549Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2253999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2254426Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2254798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2255175Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2255665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2256088Z layer_outputs = layer_module( 2025-10-10T00:56:44.2256466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2256859Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2257293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2257737Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2258167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2258598Z self_outputs = self.self( 2025-10-10T00:56:44.2259013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2259520Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2259727Z 2025-10-10T00:56:44.2259847Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2260232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2260579Z res = mod(**inputs) 2025-10-10T00:56:44.2260930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2261313Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2261737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2262166Z outputs = self.layoutlm( 2025-10-10T00:56:44.2262528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2262911Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2263341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2263768Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2264143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2264521Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2264951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2265382Z layer_outputs = layer_module( 2025-10-10T00:56:44.2265761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2266164Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2266701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2267161Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2267605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2268078Z self_outputs = self.self( 2025-10-10T00:56:44.2268510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2269029Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2269245Z 2025-10-10T00:56:44.2269344Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2269576Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2269836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2270227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2270605Z res = mod(**inputs) 2025-10-10T00:56:44.2271000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2271385Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2271819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2272249Z outputs = self.layoutlm( 2025-10-10T00:56:44.2272606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2272960Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2273386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2273823Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2274176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2274536Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2274934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2275343Z layer_outputs = layer_module( 2025-10-10T00:56:44.2275699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2276065Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2276469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2276887Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2277319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2277806Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2278273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2278687Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2278839Z 2025-10-10T00:56:44.2278947Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2279313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2279645Z res = mod(**inputs) 2025-10-10T00:56:44.2279975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2280327Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2280736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2281139Z outputs = self.layoutlm( 2025-10-10T00:56:44.2281487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2281841Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2282246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2282672Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2283028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2283388Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2283796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2284201Z layer_outputs = layer_module( 2025-10-10T00:56:44.2284559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2284928Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2285388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2285807Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2286221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2286623Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2287063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2287542Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2287999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2288419Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2288564Z 2025-10-10T00:56:44.2288687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2289088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2289449Z res = mod(**inputs) 2025-10-10T00:56:44.2289803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2290189Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2290647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2291053Z outputs = self.layoutlm( 2025-10-10T00:56:44.2291419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2291808Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2292242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2292677Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2293046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2293437Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2293864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2294301Z layer_outputs = layer_module( 2025-10-10T00:56:44.2294678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2295073Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2295512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2295980Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2296418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2296845Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2297334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2297865Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2298354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2298838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2299264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2299639Z return self.act(input) 2025-10-10T00:56:44.2299765Z 2025-10-10T00:56:44.2299876Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2300299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2300631Z res = mod(**inputs) 2025-10-10T00:56:44.2300961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2301324Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2301730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2302136Z outputs = self.layoutlm( 2025-10-10T00:56:44.2302475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2302852Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2303283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2303721Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2304095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2304468Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2304898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2305330Z layer_outputs = layer_module( 2025-10-10T00:56:44.2305710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2306102Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2306607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2307056Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2307502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2307938Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2308378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2308871Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2309356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2309457Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2309461Z 2025-10-10T00:56:44.2309575Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2309791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2309873Z res = mod(**inputs) 2025-10-10T00:56:44.2310109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2310198Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2310484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2310584Z outputs = self.layoutlm( 2025-10-10T00:56:44.2310826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2310906Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2311196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2311276Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2311514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2311593Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2311932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2312022Z layer_outputs = layer_module( 2025-10-10T00:56:44.2312264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2312356Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2312640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2312730Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2313027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2313103Z self_outputs = self.self( 2025-10-10T00:56:44.2313396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2313557Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2313561Z 2025-10-10T00:56:44.2313683Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2313898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2313968Z res = mod(**inputs) 2025-10-10T00:56:44.2314210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2314290Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2314583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2314660Z outputs = self.layoutlm( 2025-10-10T00:56:44.2314893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2314986Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2315273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2315363Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2315593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2315672Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2315964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2316043Z layer_outputs = layer_module( 2025-10-10T00:56:44.2316288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2316376Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2316671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2316764Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2317048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2317157Z self_outputs = self.self( 2025-10-10T00:56:44.2317441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2317598Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2317602Z 2025-10-10T00:56:44.2317715Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2317931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2318009Z res = mod(**inputs) 2025-10-10T00:56:44.2318285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2318390Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2318678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2318762Z outputs = self.layoutlm( 2025-10-10T00:56:44.2318996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2319076Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2319376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2319455Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2319699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2319778Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2320075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2320161Z layer_outputs = layer_module( 2025-10-10T00:56:44.2320406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2320499Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2320787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2320876Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2321172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2321248Z self_outputs = self.self( 2025-10-10T00:56:44.2321545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2321706Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2321710Z 2025-10-10T00:56:44.2321807Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2321895Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2322007Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2322232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2322302Z res = mod(**inputs) 2025-10-10T00:56:44.2322544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2322626Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2322918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2323002Z outputs = self.layoutlm( 2025-10-10T00:56:44.2323241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2323332Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2323622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2323721Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2323960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2324042Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2324332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2324410Z layer_outputs = layer_module( 2025-10-10T00:56:44.2324654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2324791Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2325081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2325181Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2325467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2325613Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2325898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2325989Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2326001Z 2025-10-10T00:56:44.2326113Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2326327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2326406Z res = mod(**inputs) 2025-10-10T00:56:44.2326638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2326723Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2327011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2327084Z outputs = self.layoutlm( 2025-10-10T00:56:44.2327321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2327399Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2327687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2327767Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2328001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2328089Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2328372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2328457Z layer_outputs = layer_module( 2025-10-10T00:56:44.2328694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2328779Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2329070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2329164Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2329451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2329535Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2329862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2329995Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2330325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2330424Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2330428Z 2025-10-10T00:56:44.2330538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2330760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2330831Z res = mod(**inputs) 2025-10-10T00:56:44.2331061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2331149Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2331625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2331723Z outputs = self.layoutlm( 2025-10-10T00:56:44.2331959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2332048Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2332336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2332415Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2332658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2332738Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2333095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2333179Z layer_outputs = layer_module( 2025-10-10T00:56:44.2333420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2333518Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2333803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2333902Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2334183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2334275Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2334602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2334732Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2335033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2335155Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2335393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2335468Z return self.act(input) 2025-10-10T00:56:44.2335472Z 2025-10-10T00:56:44.2335590Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2335799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2335866Z res = mod(**inputs) 2025-10-10T00:56:44.2336139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2336216Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2336498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2336573Z outputs = self.layoutlm( 2025-10-10T00:56:44.2336796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2336908Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2337184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2337266Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2337486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2337561Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2337837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2337911Z layer_outputs = layer_module( 2025-10-10T00:56:44.2338202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2338285Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2338556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2338648Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2338917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2339004Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2339307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2339450Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2339720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2339806Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2339810Z 2025-10-10T00:56:44.2339926Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2340130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2340204Z res = mod(**inputs) 2025-10-10T00:56:44.2340426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2340502Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2340776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2340848Z outputs = self.layoutlm( 2025-10-10T00:56:44.2341075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2341154Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2341430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2341505Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2341724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2341805Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2342075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2342155Z layer_outputs = layer_module( 2025-10-10T00:56:44.2342378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2342461Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2342740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2342827Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2343114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2343210Z self_outputs = self.self( 2025-10-10T00:56:44.2343491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2343655Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2343659Z 2025-10-10T00:56:44.2343769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2343987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2344058Z res = mod(**inputs) 2025-10-10T00:56:44.2344336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2344433Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2344715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2344801Z outputs = self.layoutlm( 2025-10-10T00:56:44.2345029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2345117Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2345399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2345478Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2345715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2345794Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2346090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2346167Z layer_outputs = layer_module( 2025-10-10T00:56:44.2346474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2346567Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2346853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2346952Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2347237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2347320Z self_outputs = self.self( 2025-10-10T00:56:44.2347611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2347754Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2347758Z 2025-10-10T00:56:44.2347873Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2348078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2348154Z res = mod(**inputs) 2025-10-10T00:56:44.2348378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2348462Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2348732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2348804Z outputs = self.layoutlm( 2025-10-10T00:56:44.2349033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2349116Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2349394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2349491Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2349713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2349796Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2350069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2350151Z layer_outputs = layer_module( 2025-10-10T00:56:44.2350379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2350461Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2350773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2350876Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2351176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2351250Z self_outputs = self.self( 2025-10-10T00:56:44.2351521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2351666Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2351670Z 2025-10-10T00:56:44.2351752Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2351841Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2351944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2352145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2352214Z res = mod(**inputs) 2025-10-10T00:56:44.2352428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2352512Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2352776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2352853Z outputs = self.layoutlm( 2025-10-10T00:56:44.2353066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2353139Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2353411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2353484Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2353713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2353790Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2354067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2354141Z layer_outputs = layer_module( 2025-10-10T00:56:44.2354373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2354459Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2354722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2354812Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2355074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2355204Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2355478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2355583Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2355588Z 2025-10-10T00:56:44.2355699Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2355899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2355971Z res = mod(**inputs) 2025-10-10T00:56:44.2356193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2356268Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2356543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2356614Z outputs = self.layoutlm( 2025-10-10T00:56:44.2356890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2356965Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2357235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2357315Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2357532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2357613Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2357880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2357952Z layer_outputs = layer_module( 2025-10-10T00:56:44.2358181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2358266Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2358539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2358630Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2358897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2358975Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2359275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2359405Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2359671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2359760Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2359768Z 2025-10-10T00:56:44.2359875Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2360079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2360149Z res = mod(**inputs) 2025-10-10T00:56:44.2360375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2360457Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2360719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2360796Z outputs = self.layoutlm( 2025-10-10T00:56:44.2361006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2361078Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2361349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2361423Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2361643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2361735Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2361995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2362074Z layer_outputs = layer_module( 2025-10-10T00:56:44.2362294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2362382Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2362644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2362734Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2363036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2363115Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2363421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2363540Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2363809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2363922Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2364134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2364212Z return self.act(input) 2025-10-10T00:56:44.2364215Z 2025-10-10T00:56:44.2364321Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2364524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2364589Z res = mod(**inputs) 2025-10-10T00:56:44.2364803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2364886Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2365150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2365228Z outputs = self.layoutlm( 2025-10-10T00:56:44.2365448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2365532Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2365800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2365879Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2366103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2366181Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2366456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2366528Z layer_outputs = layer_module( 2025-10-10T00:56:44.2366753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2366856Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2367121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2367212Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2367475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2367561Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2367862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2368024Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2368299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2368385Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2368388Z 2025-10-10T00:56:44.2368501Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2368701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2368768Z res = mod(**inputs) 2025-10-10T00:56:44.2369031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2369125Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2369399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2369473Z outputs = self.layoutlm( 2025-10-10T00:56:44.2369698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2369775Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2370042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2370126Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2370344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2370425Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2370697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2370771Z layer_outputs = layer_module( 2025-10-10T00:56:44.2371001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2371081Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2371354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2371439Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2371707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2371786Z self_outputs = self.self( 2025-10-10T00:56:44.2372057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2372214Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2372217Z 2025-10-10T00:56:44.2372481Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2372687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2372753Z res = mod(**inputs) 2025-10-10T00:56:44.2372971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2373055Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2373320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2373398Z outputs = self.layoutlm( 2025-10-10T00:56:44.2373614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2373692Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2373969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2374062Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2374289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2374364Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2374637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2374710Z layer_outputs = layer_module( 2025-10-10T00:56:44.2374941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2375037Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2376149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2376288Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2376586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2376676Z self_outputs = self.self( 2025-10-10T00:56:44.2376954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2377094Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2377098Z 2025-10-10T00:56:44.2377209Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2377410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2377483Z res = mod(**inputs) 2025-10-10T00:56:44.2377705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2377782Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2378058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2378130Z outputs = self.layoutlm( 2025-10-10T00:56:44.2378357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2378433Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2378702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2378785Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2379004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2379085Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2379357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2379438Z layer_outputs = layer_module( 2025-10-10T00:56:44.2379662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2379744Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2380035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2380123Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2380423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2380499Z self_outputs = self.self( 2025-10-10T00:56:44.2380795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2380960Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2380964Z 2025-10-10T00:56:44.2381074Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2381168Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2381279Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2381504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2381582Z res = mod(**inputs) 2025-10-10T00:56:44.2381823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2381909Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2382205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2382286Z outputs = self.layoutlm( 2025-10-10T00:56:44.2382573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2382655Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2382959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2383039Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2383287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2383367Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2383702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2383787Z layer_outputs = layer_module( 2025-10-10T00:56:44.2384027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2384123Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2384410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2384500Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2384792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2384934Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2385239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2385330Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2385334Z 2025-10-10T00:56:44.2385453Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2385668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2385743Z res = mod(**inputs) 2025-10-10T00:56:44.2385986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2386067Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2386581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2386664Z outputs = self.layoutlm( 2025-10-10T00:56:44.2386906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2386997Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2387290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2387380Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2387631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2387719Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2388004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2388101Z layer_outputs = layer_module( 2025-10-10T00:56:44.2388348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2388434Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2388737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2388829Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2389108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2389199Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2389577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2389717Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2390003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2390098Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2390102Z 2025-10-10T00:56:44.2390214Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2390424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2390503Z res = mod(**inputs) 2025-10-10T00:56:44.2390734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2390818Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2391103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2391177Z outputs = self.layoutlm( 2025-10-10T00:56:44.2391417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2391497Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2391797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2391876Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2392107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2392193Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2392492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2392582Z layer_outputs = layer_module( 2025-10-10T00:56:44.2392817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2392912Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2393196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2393289Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2393572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2393655Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2393982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2394111Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2394400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2394529Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2394777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2394861Z return self.act(input) 2025-10-10T00:56:44.2394865Z 2025-10-10T00:56:44.2394976Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2395198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2395269Z res = mod(**inputs) 2025-10-10T00:56:44.2395505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2395596Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2395916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2396018Z outputs = self.layoutlm( 2025-10-10T00:56:44.2396252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2396336Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2396628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2396708Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2396948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2397029Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2397312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2397398Z layer_outputs = layer_module( 2025-10-10T00:56:44.2397636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2397731Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2398015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2398113Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2398387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2398468Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2398795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2398939Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2399232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2399323Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2399328Z 2025-10-10T00:56:44.2399445Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2399659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2399731Z res = mod(**inputs) 2025-10-10T00:56:44.2399969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2400048Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2400336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2400412Z outputs = self.layoutlm( 2025-10-10T00:56:44.2400647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2400735Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2401020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2401124Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2401359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2401438Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2401734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2401813Z layer_outputs = layer_module( 2025-10-10T00:56:44.2402060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2402146Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2402487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2402582Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2402879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2402966Z self_outputs = self.self( 2025-10-10T00:56:44.2403260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2403432Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2403437Z 2025-10-10T00:56:44.2403551Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2403772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2403854Z res = mod(**inputs) 2025-10-10T00:56:44.2404103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2404192Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2404497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2404574Z outputs = self.layoutlm( 2025-10-10T00:56:44.2404814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2404893Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2405198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2405278Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2405519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2405599Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2405892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2405978Z layer_outputs = layer_module( 2025-10-10T00:56:44.2406217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2406308Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2406613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2406703Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2407014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2407090Z self_outputs = self.self( 2025-10-10T00:56:44.2407395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2407547Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2407550Z 2025-10-10T00:56:44.2407688Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2407901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2407971Z res = mod(**inputs) 2025-10-10T00:56:44.2408210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2408291Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2408594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2408668Z outputs = self.layoutlm( 2025-10-10T00:56:44.2408901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2409045Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2409331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2409420Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2409651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2409730Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2410036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2410115Z layer_outputs = layer_module( 2025-10-10T00:56:44.2410357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2410445Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2410749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2410840Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2411125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2411210Z self_outputs = self.self( 2025-10-10T00:56:44.2411510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2411674Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2411678Z 2025-10-10T00:56:44.2411767Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2411855Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2411973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2412191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2412268Z res = mod(**inputs) 2025-10-10T00:56:44.2412502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2412591Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2412875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2412951Z outputs = self.layoutlm( 2025-10-10T00:56:44.2413190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2413271Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2413573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2413652Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2413886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2413986Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2414268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2414373Z layer_outputs = layer_module( 2025-10-10T00:56:44.2414611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2414697Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2415042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2415131Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2415422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2415620Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2415914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2416008Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2416012Z 2025-10-10T00:56:44.2416124Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2416342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2416410Z res = mod(**inputs) 2025-10-10T00:56:44.2416651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2416731Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2417018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2417101Z outputs = self.layoutlm( 2025-10-10T00:56:44.2417337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2417424Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2417713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2417799Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2418031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2418110Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2418406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2418484Z layer_outputs = layer_module( 2025-10-10T00:56:44.2418727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2418818Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2419105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2419208Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2419488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2419579Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2419903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2420032Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2420334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2420429Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2420434Z 2025-10-10T00:56:44.2420557Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2420778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2420879Z res = mod(**inputs) 2025-10-10T00:56:44.2421118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2421200Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2421501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2421577Z outputs = self.layoutlm( 2025-10-10T00:56:44.2421823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2421905Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2422231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2422338Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2422576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2422667Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2422958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2423043Z layer_outputs = layer_module( 2025-10-10T00:56:44.2423287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2423375Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2423676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2423774Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2424070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2424157Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2424485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2424624Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2424918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2425049Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2425283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2425366Z return self.act(input) 2025-10-10T00:56:44.2425370Z 2025-10-10T00:56:44.2425489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2425707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2425788Z res = mod(**inputs) 2025-10-10T00:56:44.2426025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2426115Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2426497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2426581Z outputs = self.layoutlm( 2025-10-10T00:56:44.2426832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2426916Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2427225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2427309Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2427556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2427665Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2427947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2428035Z layer_outputs = layer_module( 2025-10-10T00:56:44.2428274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2428371Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2428652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2428745Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2429115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2429201Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2429528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2429673Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2429958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2430055Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2430059Z 2025-10-10T00:56:44.2430171Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2430393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2430463Z res = mod(**inputs) 2025-10-10T00:56:44.2430711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2430792Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2431079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2431162Z outputs = self.layoutlm( 2025-10-10T00:56:44.2431393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2431616Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2431913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2431994Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2432236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2432323Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2432618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2432698Z layer_outputs = layer_module( 2025-10-10T00:56:44.2432936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2433031Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2433315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2433416Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2433703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2433788Z self_outputs = self.self( 2025-10-10T00:56:44.2434079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2434242Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2434290Z 2025-10-10T00:56:44.2434414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2434630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2434707Z res = mod(**inputs) 2025-10-10T00:56:44.2434943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2435026Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2435326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2435402Z outputs = self.layoutlm( 2025-10-10T00:56:44.2435699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2435805Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2436101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2436184Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2436416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2436505Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2436790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2436875Z layer_outputs = layer_module( 2025-10-10T00:56:44.2437112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2437197Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2437497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2437587Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2453720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2453939Z self_outputs = self.self( 2025-10-10T00:56:44.2454286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2454438Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2454445Z 2025-10-10T00:56:44.2454578Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2454798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2454877Z res = mod(**inputs) 2025-10-10T00:56:44.2455165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2455251Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2455548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2455633Z outputs = self.layoutlm( 2025-10-10T00:56:44.2455876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2455954Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2456223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2456312Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2456530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2456621Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2456888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2457078Z layer_outputs = layer_module( 2025-10-10T00:56:44.2457303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2457390Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2457667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2457755Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2458029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2458107Z self_outputs = self.self( 2025-10-10T00:56:44.2458441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2458628Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2458635Z 2025-10-10T00:56:44.2458721Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2458813Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2458923Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2459137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2459209Z res = mod(**inputs) 2025-10-10T00:56:44.2459429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2459516Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2459782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2459867Z outputs = self.layoutlm( 2025-10-10T00:56:44.2460086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2460162Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2460434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2460511Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2460732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2460806Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2461067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2461151Z layer_outputs = layer_module( 2025-10-10T00:56:44.2461379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2461474Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2461744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2461839Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2462105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2462240Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2462517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2462604Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2462608Z 2025-10-10T00:56:44.2462726Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2462937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2463007Z res = mod(**inputs) 2025-10-10T00:56:44.2463235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2463331Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2463607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2463686Z outputs = self.layoutlm( 2025-10-10T00:56:44.2463926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2464007Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2464290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2464381Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2464647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2464755Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2465037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2465118Z layer_outputs = layer_module( 2025-10-10T00:56:44.2465365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2465451Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2465744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2465839Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2466121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2466219Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2466645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2466803Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2467099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2467202Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2467207Z 2025-10-10T00:56:44.2467325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2467551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2467640Z res = mod(**inputs) 2025-10-10T00:56:44.2467857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2467943Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2468206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2468279Z outputs = self.layoutlm( 2025-10-10T00:56:44.2468505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2468580Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2468870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2468951Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2469191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2469273Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2469564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2469656Z layer_outputs = layer_module( 2025-10-10T00:56:44.2469896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2470015Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2470301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2470394Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2470685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2470768Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2471097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2471262Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2471575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2471702Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2471934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2472023Z return self.act(input) 2025-10-10T00:56:44.2472027Z 2025-10-10T00:56:44.2472141Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2472363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2472435Z res = mod(**inputs) 2025-10-10T00:56:44.2472672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2472765Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2473060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2473146Z outputs = self.layoutlm( 2025-10-10T00:56:44.2473382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2473465Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2473762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2473842Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2474082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2474163Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2474458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2474541Z layer_outputs = layer_module( 2025-10-10T00:56:44.2474780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2474880Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2475167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2475268Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2475546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2475631Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2475962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2476109Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2476411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2476503Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2476526Z 2025-10-10T00:56:44.2476650Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2476848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2476914Z res = mod(**inputs) 2025-10-10T00:56:44.2477135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2477210Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2477479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2477550Z outputs = self.layoutlm( 2025-10-10T00:56:44.2477790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2477890Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2478150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2478233Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2478444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2478518Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2478787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2478860Z layer_outputs = layer_module( 2025-10-10T00:56:44.2479083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2479163Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2479435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2479520Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2479783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2479863Z self_outputs = self.self( 2025-10-10T00:56:44.2480125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-10-10T00:56:44.2480281Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2480285Z 2025-10-10T00:56:44.2480388Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2480591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2480658Z res = mod(**inputs) 2025-10-10T00:56:44.2480878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2480964Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2481227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2481307Z outputs = self.layoutlm( 2025-10-10T00:56:44.2481521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2481597Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2481867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2481941Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2482161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2482241Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2482504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2482612Z layer_outputs = layer_module( 2025-10-10T00:56:44.2482834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2482923Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2483190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2483287Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2483569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2483647Z self_outputs = self.self( 2025-10-10T00:56:44.2483970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-10-10T00:56:44.2484137Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2484143Z 2025-10-10T00:56:44.2484261Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2484474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2484559Z res = mod(**inputs) 2025-10-10T00:56:44.2484788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2484865Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2485140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2485212Z outputs = self.layoutlm( 2025-10-10T00:56:44.2485433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2485517Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2485785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2485870Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2486088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2486172Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2486439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2486524Z layer_outputs = layer_module( 2025-10-10T00:56:44.2486751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2486830Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2487104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2487187Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2487453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-10-10T00:56:44.2487532Z self_outputs = self.self( 2025-10-10T00:56:44.2487797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-10-10T00:56:44.2487955Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-10-10T00:56:44.2487959Z 2025-10-10T00:56:44.2488044Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2488138Z cudagraph partition due to non gpu ops 2025-10-10T00:56:44.2488245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2488451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2488531Z res = mod(**inputs) 2025-10-10T00:56:44.2488751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2488853Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2489131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2489203Z outputs = self.layoutlm( 2025-10-10T00:56:44.2489425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2489503Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2489781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2489857Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2490130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2490218Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2490483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2490569Z layer_outputs = layer_module( 2025-10-10T00:56:44.2490801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2490887Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2491148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-10-10T00:56:44.2491230Z self_attention_outputs = self.attention( 2025-10-10T00:56:44.2491502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-10-10T00:56:44.2491637Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:56:44.2491910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-10-10T00:56:44.2491996Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2491999Z 2025-10-10T00:56:44.2492102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2492304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2492371Z res = mod(**inputs) 2025-10-10T00:56:44.2492594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2492670Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2492944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2493020Z outputs = self.layoutlm( 2025-10-10T00:56:44.2493238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2493324Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2493594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2493679Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2493898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2493975Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2494252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2494327Z layer_outputs = layer_module( 2025-10-10T00:56:44.2494561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2494645Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2494914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2495028Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2495291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2495379Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2495685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2495818Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2496089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-10-10T00:56:44.2496219Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2496224Z 2025-10-10T00:56:44.2496341Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2496541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2496621Z res = mod(**inputs) 2025-10-10T00:56:44.2496846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2496924Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2497212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2497285Z outputs = self.layoutlm( 2025-10-10T00:56:44.2497519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2497600Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2497905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2497984Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2498221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2498314Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2498608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2498697Z layer_outputs = layer_module( 2025-10-10T00:56:44.2498938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2499025Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2499325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2499421Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2499715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2499795Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2500110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-10-10T00:56:44.2500243Z intermediate_output = self.intermediate(attention_output) 2025-10-10T00:56:44.2500520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-10-10T00:56:44.2500650Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:56:44.2500887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:56:44.2500974Z return self.act(input) 2025-10-10T00:56:44.2500980Z 2025-10-10T00:56:44.2501095Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2501314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2501414Z res = mod(**inputs) 2025-10-10T00:56:44.2501650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2501740Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2502028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-10-10T00:56:44.2502105Z outputs = self.layoutlm( 2025-10-10T00:56:44.2502355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2502436Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2502765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-10-10T00:56:44.2502872Z encoder_outputs = self.encoder( 2025-10-10T00:56:44.2503116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2503200Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2503489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-10-10T00:56:44.2503577Z layer_outputs = layer_module( 2025-10-10T00:56:44.2503815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:56:44.2503910Z return super().__call__(*args, **kwargs) 2025-10-10T00:56:44.2504200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-10-10T00:56:44.2504293Z layer_output = apply_chunking_to_forward( 2025-10-10T00:56:44.2504590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:56:44.2504677Z return forward_fn(*input_tensors) 2025-10-10T00:56:44.2505011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-10-10T00:56:44.2505159Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:56:44.2505455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-10-10T00:56:44.2505547Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2505551Z 2025-10-10T00:56:44.2505665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2505890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2505962Z res = mod(**inputs) 2025-10-10T00:56:44.2506212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2506371Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2506674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 736, in forward 2025-10-10T00:56:44.2506793Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:56:44.2507093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 457, in forward 2025-10-10T00:56:44.2507241Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:56:44.2507532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 445, in forward 2025-10-10T00:56:44.2507646Z hidden_states = self.transform(hidden_states) 2025-10-10T00:56:44.2507937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 420, in forward 2025-10-10T00:56:44.2508031Z hidden_states = self.dense(hidden_states) 2025-10-10T00:56:44.2508035Z 2025-10-10T00:56:44.2508179Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2508404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2508485Z res = mod(**inputs) 2025-10-10T00:56:44.2508724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2508806Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2509101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 736, in forward 2025-10-10T00:56:44.2509203Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:56:44.2509532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 457, in forward 2025-10-10T00:56:44.2509670Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:56:44.2509967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 446, in forward 2025-10-10T00:56:44.2510071Z hidden_states = self.decoder(hidden_states) 2025-10-10T00:56:44.2510075Z 2025-10-10T00:56:44.2510190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:56:44.2510418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:56:44.2510491Z res = mod(**inputs) 2025-10-10T00:56:44.2510742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T00:56:44.2510827Z output = func(self, *args, **kwargs) 2025-10-10T00:56:44.2511124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 741, in forward 2025-10-10T00:56:44.2511217Z masked_lm_loss = loss_fct( 2025-10-10T00:56:44.2511221Z 2025-10-10T00:56:54.8107617Z Compilation time (from dynamo_timed): 17.934275536 2025-10-10T00:56:54.8171929Z pass 2025-10-10T00:56:54.8172542Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:56:54.8173564Z TIMING: _recursive_pre_grad_passes:0.00841 _recursive_joint_graph_passes:0.50521 _recursive_post_grad_passes:0.07593 async_compile.wait:0.64466 code_gen:9.78295 inductor_compile:11.12476 backend_compile:14.91559 gc:0.00012 entire_frame_compile:17.93428 total_wall_time:17.93428 2025-10-10T00:56:54.8174697Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:8952 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:2595 2025-10-10T00:56:54.8175372Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-10-10T00:56:57.8203012Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:56:57.8204866Z import pynvml # type: ignore[import] 2025-10-10T00:57:01.2419512Z 2025-10-10T00:57:08.0877119Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:57:08.0877603Z loading model: 0it [00:06, ?it/s] 2025-10-10T00:57:08.0906282Z cpu eval M2M100ForConditionalGeneration 2025-10-10T00:57:08.9107400Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:09.2794401Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:09.6805686Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:26.7119805Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7120394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7120787Z res = mod(**inputs) 2025-10-10T00:57:26.7121291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7122116Z outputs = self.model( 2025-10-10T00:57:26.7122555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7123027Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7123468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-10-10T00:57:26.7123949Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-10-10T00:57:26.7124491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7125072Z return func(*args, **kwargs) 2025-10-10T00:57:26.7125536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7126146Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7126845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7127364Z mask = input_ids.ne(padding_idx).int() 2025-10-10T00:57:26.7127551Z 2025-10-10T00:57:26.7127673Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7128077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7128447Z res = mod(**inputs) 2025-10-10T00:57:26.7128856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7129289Z outputs = self.model( 2025-10-10T00:57:26.7129711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7130159Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7130598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-10-10T00:57:26.7131147Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-10-10T00:57:26.7131979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7132407Z return func(*args, **kwargs) 2025-10-10T00:57:26.7132837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7133418Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7134069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7134569Z mask = input_ids.ne(padding_idx).int() 2025-10-10T00:57:26.7134730Z 2025-10-10T00:57:26.7134822Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7135064Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7135291Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7135521Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7135747Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7135989Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7136207Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7136433Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7136655Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7136887Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7137103Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7137334Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7137593Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7138030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7138452Z res = mod(**inputs) 2025-10-10T00:57:26.7138852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7139282Z outputs = self.model( 2025-10-10T00:57:26.7139695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7140140Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7140554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-10-10T00:57:26.7141113Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-10-10T00:57:26.7141571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7141979Z return func(*args, **kwargs) 2025-10-10T00:57:26.7142414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7142992Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7143637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7144262Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T00:57:26.7144542Z 2025-10-10T00:57:26.7144675Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7145097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7145461Z res = mod(**inputs) 2025-10-10T00:57:26.7145874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7146523Z outputs = self.model( 2025-10-10T00:57:26.7146955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7147391Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7147811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-10-10T00:57:26.7148296Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-10-10T00:57:26.7148756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7149173Z return func(*args, **kwargs) 2025-10-10T00:57:26.7149597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7150179Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7150821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7151447Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T00:57:26.7151718Z 2025-10-10T00:57:26.7151840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7152243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7152620Z res = mod(**inputs) 2025-10-10T00:57:26.7153049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7153479Z outputs = self.model( 2025-10-10T00:57:26.7153926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7154368Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7154798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7155227Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7155634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7156055Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7156501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7157013Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7157472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7157881Z return func(*args, **kwargs) 2025-10-10T00:57:26.7158285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7158780Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7159000Z 2025-10-10T00:57:26.7159114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7159502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7159847Z res = mod(**inputs) 2025-10-10T00:57:26.7160242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7160657Z outputs = self.model( 2025-10-10T00:57:26.7161044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7161463Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7161872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7162285Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7162660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7163058Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7163476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7163913Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7164346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7164743Z return func(*args, **kwargs) 2025-10-10T00:57:26.7165144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7165574Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7165726Z 2025-10-10T00:57:26.7165849Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7166238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7166580Z res = mod(**inputs) 2025-10-10T00:57:26.7166972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7167384Z outputs = self.model( 2025-10-10T00:57:26.7167781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7168195Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7168597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7169040Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7169421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7169814Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7170229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7170667Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7171091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7171498Z return func(*args, **kwargs) 2025-10-10T00:57:26.7171978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7172405Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7172571Z 2025-10-10T00:57:26.7172660Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7172928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7173330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7173702Z res = mod(**inputs) 2025-10-10T00:57:26.7174102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7174528Z outputs = self.model( 2025-10-10T00:57:26.7174935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7175370Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7175799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7176231Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7176626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7177033Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7177463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7177915Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7178354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7178778Z return func(*args, **kwargs) 2025-10-10T00:57:26.7179195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7179646Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7180152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7180696Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7180911Z 2025-10-10T00:57:26.7181030Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7181434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7181804Z res = mod(**inputs) 2025-10-10T00:57:26.7182213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7182639Z outputs = self.model( 2025-10-10T00:57:26.7183047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7183491Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7183952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7184434Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7184835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7185245Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7185682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7186143Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7186674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7187105Z return func(*args, **kwargs) 2025-10-10T00:57:26.7187595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7188028Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7188198Z 2025-10-10T00:57:26.7188316Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7188720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7189077Z res = mod(**inputs) 2025-10-10T00:57:26.7189479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7189897Z outputs = self.model( 2025-10-10T00:57:26.7190303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7190735Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7191161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7191598Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7191981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7192388Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7192820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7193300Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7193493Z 2025-10-10T00:57:26.7193610Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7194005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7194361Z res = mod(**inputs) 2025-10-10T00:57:26.7194764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7195190Z outputs = self.model( 2025-10-10T00:57:26.7195588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7196019Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7196448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7196861Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7197232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7197624Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7198043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7198515Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7198701Z 2025-10-10T00:57:26.7198823Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7199203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7200296Z res = mod(**inputs) 2025-10-10T00:57:26.7200697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7201125Z outputs = self.model( 2025-10-10T00:57:26.7201524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7201942Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7202373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7202821Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7203261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7203695Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7204130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7204581Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7204745Z 2025-10-10T00:57:26.7204860Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7205253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7205602Z res = mod(**inputs) 2025-10-10T00:57:26.7206005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7206427Z outputs = self.model( 2025-10-10T00:57:26.7206834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7207264Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7207678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7208116Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7208510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7208930Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7209364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7209817Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7210263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7210687Z return func(*args, **kwargs) 2025-10-10T00:57:26.7211109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7211622Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7211856Z 2025-10-10T00:57:26.7211972Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7212387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7212744Z res = mod(**inputs) 2025-10-10T00:57:26.7213143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7213557Z outputs = self.model( 2025-10-10T00:57:26.7213960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7214398Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7214825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7215253Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7215633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7216064Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7216495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7216947Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7217382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7217805Z return func(*args, **kwargs) 2025-10-10T00:57:26.7218222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7218723Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7218878Z 2025-10-10T00:57:26.7219002Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7219398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7219769Z res = mod(**inputs) 2025-10-10T00:57:26.7220194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7220617Z outputs = self.model( 2025-10-10T00:57:26.7221036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7221471Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7221902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7222347Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7222741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7223137Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7223576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7224026Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7224467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7224883Z return func(*args, **kwargs) 2025-10-10T00:57:26.7225294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7225741Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7225905Z 2025-10-10T00:57:26.7225998Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7226275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7226788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7227148Z res = mod(**inputs) 2025-10-10T00:57:26.7227558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7227987Z outputs = self.model( 2025-10-10T00:57:26.7228398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7228822Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7229246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7229671Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7230067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7230476Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7230901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7231384Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7231982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7232406Z return func(*args, **kwargs) 2025-10-10T00:57:26.7232825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7233273Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7233771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7234423Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7234665Z 2025-10-10T00:57:26.7234792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7235194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7235554Z res = mod(**inputs) 2025-10-10T00:57:26.7235963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7236397Z outputs = self.model( 2025-10-10T00:57:26.7236822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7237249Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7237684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7238123Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7238523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7238933Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7239361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7239812Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7240250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7240668Z return func(*args, **kwargs) 2025-10-10T00:57:26.7241072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7241492Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7241650Z 2025-10-10T00:57:26.7241763Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7242156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7242508Z res = mod(**inputs) 2025-10-10T00:57:26.7242904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7243340Z outputs = self.model( 2025-10-10T00:57:26.7243730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7244159Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7244571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7244992Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7245387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7245786Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7246220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7246730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7246920Z 2025-10-10T00:57:26.7247035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7247436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7247804Z res = mod(**inputs) 2025-10-10T00:57:26.7248195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7248598Z outputs = self.model( 2025-10-10T00:57:26.7248990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7249407Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7249877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7250292Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7250666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7251064Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7251496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7251976Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7252161Z 2025-10-10T00:57:26.7252281Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7252688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7253041Z res = mod(**inputs) 2025-10-10T00:57:26.7253451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7253882Z outputs = self.model( 2025-10-10T00:57:26.7254271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7254704Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7255137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7255574Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7255966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7256374Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7256810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7257257Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7257411Z 2025-10-10T00:57:26.7257535Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7257934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7258287Z res = mod(**inputs) 2025-10-10T00:57:26.7258687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7259116Z outputs = self.model( 2025-10-10T00:57:26.7259537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7259961Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7260383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7260808Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7261200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7261600Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7262045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7262496Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7262660Z 2025-10-10T00:57:26.7262778Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7263273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7263651Z res = mod(**inputs) 2025-10-10T00:57:26.7264047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7264479Z outputs = self.model( 2025-10-10T00:57:26.7264980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7265438Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7265865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7266374Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7266775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7267179Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7267609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7268056Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7268502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7268932Z return func(*args, **kwargs) 2025-10-10T00:57:26.7269350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7269869Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7270101Z 2025-10-10T00:57:26.7270219Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7270620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7270979Z res = mod(**inputs) 2025-10-10T00:57:26.7271381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7271798Z outputs = self.model( 2025-10-10T00:57:26.7272201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7272632Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7273065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7273490Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7273874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7274269Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7274697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7275127Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7275552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7275948Z return func(*args, **kwargs) 2025-10-10T00:57:26.7276352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7276775Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7276918Z 2025-10-10T00:57:26.7277039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7277452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7277795Z res = mod(**inputs) 2025-10-10T00:57:26.7278188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7278603Z outputs = self.model( 2025-10-10T00:57:26.7278998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7279411Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7279834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7280303Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7280683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7281079Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7281494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7281926Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7282347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7282755Z return func(*args, **kwargs) 2025-10-10T00:57:26.7283151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7283580Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7283743Z 2025-10-10T00:57:26.7283838Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7284103Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7284489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7284831Z res = mod(**inputs) 2025-10-10T00:57:26.7285226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7285639Z outputs = self.model( 2025-10-10T00:57:26.7286034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7286453Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7286860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7287276Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7287667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7288059Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7289266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7289704Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7290131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7290544Z return func(*args, **kwargs) 2025-10-10T00:57:26.7290959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7291405Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7291884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7292423Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7292627Z 2025-10-10T00:57:26.7292751Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7293188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7293539Z res = mod(**inputs) 2025-10-10T00:57:26.7293942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7294365Z outputs = self.model( 2025-10-10T00:57:26.7294767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7295191Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7295601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7296081Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7296476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7296879Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7297302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7297746Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7298183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7298616Z return func(*args, **kwargs) 2025-10-10T00:57:26.7299027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7299481Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7299641Z 2025-10-10T00:57:26.7299762Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7300155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7300543Z res = mod(**inputs) 2025-10-10T00:57:26.7300948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7301367Z outputs = self.model( 2025-10-10T00:57:26.7301771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7302225Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7302651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7303091Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7303482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7303889Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7304321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7304804Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7304995Z 2025-10-10T00:57:26.7305110Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7305502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7305856Z res = mod(**inputs) 2025-10-10T00:57:26.7306257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7306779Z outputs = self.model( 2025-10-10T00:57:26.7307204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7307650Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7308084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7308541Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7308925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7309332Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7309764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7310238Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7310431Z 2025-10-10T00:57:26.7310554Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7310945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7311362Z res = mod(**inputs) 2025-10-10T00:57:26.7311768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7312198Z outputs = self.model( 2025-10-10T00:57:26.7312601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7313022Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7313441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7313864Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7314252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7314646Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7315077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7315513Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7315663Z 2025-10-10T00:57:26.7315785Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7316181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7316530Z res = mod(**inputs) 2025-10-10T00:57:26.7316932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7317351Z outputs = self.model( 2025-10-10T00:57:26.7317753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7318178Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7318590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7319020Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7319421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7319815Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7320228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7320676Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7321103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7321520Z return func(*args, **kwargs) 2025-10-10T00:57:26.7321921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7322422Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7322652Z 2025-10-10T00:57:26.7322770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7323155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7323528Z res = mod(**inputs) 2025-10-10T00:57:26.7323920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7324323Z outputs = self.model( 2025-10-10T00:57:26.7324717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7325150Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7325573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7326004Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7326433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7326847Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7327267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7327720Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7328154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7328583Z return func(*args, **kwargs) 2025-10-10T00:57:26.7329008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7329445Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7329593Z 2025-10-10T00:57:26.7329714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7330096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7330457Z res = mod(**inputs) 2025-10-10T00:57:26.7330850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7331266Z outputs = self.model( 2025-10-10T00:57:26.7331882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7332313Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7332727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7333142Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7333530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7333922Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7334352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7334784Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7335213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7335627Z return func(*args, **kwargs) 2025-10-10T00:57:26.7336035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7336464Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7336624Z 2025-10-10T00:57:26.7336715Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7336978Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7337361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7337710Z res = mod(**inputs) 2025-10-10T00:57:26.7338092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7338485Z outputs = self.model( 2025-10-10T00:57:26.7338914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7339306Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7339708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7340131Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7340508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7340899Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7341316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7341852Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7342282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7342689Z return func(*args, **kwargs) 2025-10-10T00:57:26.7343085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7343537Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7344020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7344540Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7344738Z 2025-10-10T00:57:26.7344860Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7345245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7345602Z res = mod(**inputs) 2025-10-10T00:57:26.7346003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7346523Z outputs = self.model( 2025-10-10T00:57:26.7346931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7347369Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7347816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7348245Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7348608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7348975Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7349380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7349796Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7350205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7350592Z return func(*args, **kwargs) 2025-10-10T00:57:26.7350968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7351371Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7351518Z 2025-10-10T00:57:26.7351626Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7351994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7352333Z res = mod(**inputs) 2025-10-10T00:57:26.7352690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7353076Z outputs = self.model( 2025-10-10T00:57:26.7353447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7353864Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7354251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7354642Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7355007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7355372Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7355760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7356182Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7356411Z 2025-10-10T00:57:26.7356519Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7356883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7357213Z res = mod(**inputs) 2025-10-10T00:57:26.7357585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7357972Z outputs = self.model( 2025-10-10T00:57:26.7358348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7358748Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7359144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7359556Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7359940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7360348Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7360768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7361231Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7361413Z 2025-10-10T00:57:26.7361527Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7361906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7362241Z res = mod(**inputs) 2025-10-10T00:57:26.7362604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7362978Z outputs = self.model( 2025-10-10T00:57:26.7363343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7363740Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7364129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7364522Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7364880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7365247Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7365631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7366025Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7366166Z 2025-10-10T00:57:26.7366278Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7366646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7366967Z res = mod(**inputs) 2025-10-10T00:57:26.7367326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7367740Z outputs = self.model( 2025-10-10T00:57:26.7368096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7368489Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7368874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7369264Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7369627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7370011Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7370455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7370862Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7371000Z 2025-10-10T00:57:26.7371112Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7371472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7371795Z res = mod(**inputs) 2025-10-10T00:57:26.7372164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7372553Z outputs = self.model( 2025-10-10T00:57:26.7372925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7373312Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7373702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7374095Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7374453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7374825Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7375225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7375658Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7376076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7376460Z return func(*args, **kwargs) 2025-10-10T00:57:26.7376837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7377300Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7377515Z 2025-10-10T00:57:26.7377624Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7377989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7378321Z res = mod(**inputs) 2025-10-10T00:57:26.7378675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7379052Z outputs = self.model( 2025-10-10T00:57:26.7379415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7379806Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7380194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7380573Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7380936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7381319Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7381766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7382212Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7382631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7383056Z return func(*args, **kwargs) 2025-10-10T00:57:26.7383462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7383893Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7384039Z 2025-10-10T00:57:26.7384158Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7384602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7384962Z res = mod(**inputs) 2025-10-10T00:57:26.7385367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7385795Z outputs = self.model( 2025-10-10T00:57:26.7386210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7386749Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7387184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7387620Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7388018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7388417Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7388845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7389288Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7389721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7390130Z return func(*args, **kwargs) 2025-10-10T00:57:26.7390537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7390977Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7391131Z 2025-10-10T00:57:26.7391229Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7391491Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7391871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7392225Z res = mod(**inputs) 2025-10-10T00:57:26.7392626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7393048Z outputs = self.model( 2025-10-10T00:57:26.7393437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7393862Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7394276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7394697Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7395078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7395467Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7395869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7396288Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7396692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7397103Z return func(*args, **kwargs) 2025-10-10T00:57:26.7397490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7397936Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7398421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7398947Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7399147Z 2025-10-10T00:57:26.7399275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7399776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7400112Z res = mod(**inputs) 2025-10-10T00:57:26.7400490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7400892Z outputs = self.model( 2025-10-10T00:57:26.7401266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7401667Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7402067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7402480Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7402862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7403262Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7403691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7404131Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7404565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7404973Z return func(*args, **kwargs) 2025-10-10T00:57:26.7405349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7405753Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7405901Z 2025-10-10T00:57:26.7406008Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7406379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7406717Z res = mod(**inputs) 2025-10-10T00:57:26.7407116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7407533Z outputs = self.model( 2025-10-10T00:57:26.7407932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7408355Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7408763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7409183Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7409549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7409928Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7410327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7410774Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7410959Z 2025-10-10T00:57:26.7411068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7411457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7411791Z res = mod(**inputs) 2025-10-10T00:57:26.7412156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7412548Z outputs = self.model( 2025-10-10T00:57:26.7412925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7413328Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7413720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7414177Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7414541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7414918Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7415331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7415806Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7415991Z 2025-10-10T00:57:26.7416105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7416511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7416871Z res = mod(**inputs) 2025-10-10T00:57:26.7417263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7417651Z outputs = self.model( 2025-10-10T00:57:26.7418059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7418478Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7418908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7419332Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7419708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7420087Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7420483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7420884Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7421023Z 2025-10-10T00:57:26.7421139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7421503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7421831Z res = mod(**inputs) 2025-10-10T00:57:26.7422201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7422598Z outputs = self.model( 2025-10-10T00:57:26.7422983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7423413Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7423823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7424247Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7424626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7425021Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7425444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7425910Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7426420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7426854Z return func(*args, **kwargs) 2025-10-10T00:57:26.7427287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7427816Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7428052Z 2025-10-10T00:57:26.7428168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7428567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7428938Z res = mod(**inputs) 2025-10-10T00:57:26.7429392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7429814Z outputs = self.model( 2025-10-10T00:57:26.7430216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7430640Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7431046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7431716Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7432104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7432517Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7432942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7433377Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7433812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7434223Z return func(*args, **kwargs) 2025-10-10T00:57:26.7434629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7435047Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7435202Z 2025-10-10T00:57:26.7435316Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7435704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7436051Z res = mod(**inputs) 2025-10-10T00:57:26.7436438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7436849Z outputs = self.model( 2025-10-10T00:57:26.7437237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7437633Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7438026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7438417Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7438765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7439138Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7439535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7439963Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7440392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7440771Z return func(*args, **kwargs) 2025-10-10T00:57:26.7441146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7441607Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7441752Z 2025-10-10T00:57:26.7441843Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7442080Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7442447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7442767Z res = mod(**inputs) 2025-10-10T00:57:26.7443130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7443506Z outputs = self.model( 2025-10-10T00:57:26.7443916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7444329Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7444708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7445092Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7445439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7445808Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7446227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7446672Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7447099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7447505Z return func(*args, **kwargs) 2025-10-10T00:57:26.7447910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7448355Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7448835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7449337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7449520Z 2025-10-10T00:57:26.7449626Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7449980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7450301Z res = mod(**inputs) 2025-10-10T00:57:26.7450668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7451059Z outputs = self.model( 2025-10-10T00:57:26.7451433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7451829Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7452218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7452641Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7453013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7453412Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7453831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7454276Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7454703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7455108Z return func(*args, **kwargs) 2025-10-10T00:57:26.7455521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7455969Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7456128Z 2025-10-10T00:57:26.7456241Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7460385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7460769Z res = mod(**inputs) 2025-10-10T00:57:26.7461177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7461606Z outputs = self.model( 2025-10-10T00:57:26.7462059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7462513Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7462940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7463382Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7463777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7464185Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7464690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7465174Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7465371Z 2025-10-10T00:57:26.7465489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7465892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7466257Z res = mod(**inputs) 2025-10-10T00:57:26.7466773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7467213Z outputs = self.model( 2025-10-10T00:57:26.7467615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7468014Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7468410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7468807Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7469170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7469535Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7469938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7470377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7470555Z 2025-10-10T00:57:26.7470671Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7471032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7471368Z res = mod(**inputs) 2025-10-10T00:57:26.7471763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7472177Z outputs = self.model( 2025-10-10T00:57:26.7472569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7472976Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7473396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7473833Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7474230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7474658Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7475073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7475502Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7475659Z 2025-10-10T00:57:26.7475845Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7476233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7476571Z res = mod(**inputs) 2025-10-10T00:57:26.7476963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7477416Z outputs = self.model( 2025-10-10T00:57:26.7477815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7478231Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7478642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7479070Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7479464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7479872Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7480304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7480734Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7480890Z 2025-10-10T00:57:26.7481007Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7481394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7481741Z res = mod(**inputs) 2025-10-10T00:57:26.7482125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7482537Z outputs = self.model( 2025-10-10T00:57:26.7482934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7483355Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7483763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7484169Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7484551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7484946Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7485363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7485818Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7486254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7486677Z return func(*args, **kwargs) 2025-10-10T00:57:26.7487091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7487598Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7487820Z 2025-10-10T00:57:26.7487935Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7488324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7488671Z res = mod(**inputs) 2025-10-10T00:57:26.7489064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7489530Z outputs = self.model( 2025-10-10T00:57:26.7489917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7490342Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7490785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7491196Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7491572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7491957Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7492397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7492847Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7493276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7493685Z return func(*args, **kwargs) 2025-10-10T00:57:26.7494085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7494511Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7494657Z 2025-10-10T00:57:26.7494778Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7495163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7495507Z res = mod(**inputs) 2025-10-10T00:57:26.7495901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7496318Z outputs = self.model( 2025-10-10T00:57:26.7496755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7497183Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7497615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7498038Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7498437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7498864Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7499286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7499722Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7500164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7500579Z return func(*args, **kwargs) 2025-10-10T00:57:26.7500992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7501423Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7501583Z 2025-10-10T00:57:26.7501676Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7501944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7502342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7502693Z res = mod(**inputs) 2025-10-10T00:57:26.7503095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7503522Z outputs = self.model( 2025-10-10T00:57:26.7503929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7504360Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7504812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7505242Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7505635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7506073Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7506613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7507083Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7507556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7507997Z return func(*args, **kwargs) 2025-10-10T00:57:26.7508426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7508882Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7509403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7509937Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7510148Z 2025-10-10T00:57:26.7510265Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7510669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7511044Z res = mod(**inputs) 2025-10-10T00:57:26.7511449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7511883Z outputs = self.model( 2025-10-10T00:57:26.7512303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7512744Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7513183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7513618Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7514012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7514419Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7514874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7515325Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7515763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7516181Z return func(*args, **kwargs) 2025-10-10T00:57:26.7516597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7517037Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7517192Z 2025-10-10T00:57:26.7517308Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7517711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7518068Z res = mod(**inputs) 2025-10-10T00:57:26.7518473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7518901Z outputs = self.model( 2025-10-10T00:57:26.7519312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7519735Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7520151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7520608Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7520989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7521376Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7521822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7522284Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7522468Z 2025-10-10T00:57:26.7522585Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7522976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7523346Z res = mod(**inputs) 2025-10-10T00:57:26.7523741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7524160Z outputs = self.model( 2025-10-10T00:57:26.7524558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7524973Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7525390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7525806Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7526190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7526584Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7526999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7527466Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7527659Z 2025-10-10T00:57:26.7527772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7528161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7528502Z res = mod(**inputs) 2025-10-10T00:57:26.7528899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7529313Z outputs = self.model( 2025-10-10T00:57:26.7529708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7530129Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7530537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7530957Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7531339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7531905Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7532351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7532778Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7532940Z 2025-10-10T00:57:26.7533054Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7533440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7533778Z res = mod(**inputs) 2025-10-10T00:57:26.7534145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7534541Z outputs = self.model( 2025-10-10T00:57:26.7534915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7535373Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7535773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7536167Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7536573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7536951Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7537350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7537766Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7538246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7538653Z return func(*args, **kwargs) 2025-10-10T00:57:26.7539064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7539537Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7539746Z 2025-10-10T00:57:26.7539861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7540228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7540580Z res = mod(**inputs) 2025-10-10T00:57:26.7540973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7541386Z outputs = self.model( 2025-10-10T00:57:26.7541777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7542197Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7542612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7542693Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7542942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7543033Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7543311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7543412Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7543674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7543765Z return func(*args, **kwargs) 2025-10-10T00:57:26.7544035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7544132Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7544136Z 2025-10-10T00:57:26.7544249Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7544473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7544544Z res = mod(**inputs) 2025-10-10T00:57:26.7544821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7544902Z outputs = self.model( 2025-10-10T00:57:26.7545177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7545264Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7545532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7545634Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7545882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7545969Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7546263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7546427Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7546697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7546785Z return func(*args, **kwargs) 2025-10-10T00:57:26.7547086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7547210Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7547214Z 2025-10-10T00:57:26.7547306Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7547433Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7547654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7547728Z res = mod(**inputs) 2025-10-10T00:57:26.7548024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7548101Z outputs = self.model( 2025-10-10T00:57:26.7548386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7548461Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7548719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7548810Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7549057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7549156Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7549434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7549538Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7549822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7549902Z return func(*args, **kwargs) 2025-10-10T00:57:26.7550187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7550303Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7550639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7550791Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7550796Z 2025-10-10T00:57:26.7550910Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7551138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7551211Z res = mod(**inputs) 2025-10-10T00:57:26.7551507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7551584Z outputs = self.model( 2025-10-10T00:57:26.7551866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7551955Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7552237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7552327Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7552591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7552686Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7552961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7553082Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7553360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7553441Z return func(*args, **kwargs) 2025-10-10T00:57:26.7553741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7553863Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7553867Z 2025-10-10T00:57:26.7553982Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7554209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7554279Z res = mod(**inputs) 2025-10-10T00:57:26.7554567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7554645Z outputs = self.model( 2025-10-10T00:57:26.7554932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7555013Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7555287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7555377Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7555621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7555717Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7555991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7556122Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7556126Z 2025-10-10T00:57:26.7556246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7556464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7556542Z res = mod(**inputs) 2025-10-10T00:57:26.7556828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7556906Z outputs = self.model( 2025-10-10T00:57:26.7557165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7557240Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7557503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7557577Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7557812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7557895Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7558146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7558276Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7558279Z 2025-10-10T00:57:26.7558384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7558593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7558659Z res = mod(**inputs) 2025-10-10T00:57:26.7558936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7559013Z outputs = self.model( 2025-10-10T00:57:26.7559271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7559373Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7559628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7559707Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7559933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7560046Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7560306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7560391Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7560395Z 2025-10-10T00:57:26.7560507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7560705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7560771Z res = mod(**inputs) 2025-10-10T00:57:26.7561037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7561107Z outputs = self.model( 2025-10-10T00:57:26.7561370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7561443Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7561706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7561779Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7562006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7562095Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7562351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7562444Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7562448Z 2025-10-10T00:57:26.7562552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7562751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7562823Z res = mod(**inputs) 2025-10-10T00:57:26.7563084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7563159Z outputs = self.model( 2025-10-10T00:57:26.7563416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7563493Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7563751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7563827Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7564059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7564141Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7564399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7564496Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7564741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7564844Z return func(*args, **kwargs) 2025-10-10T00:57:26.7565096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7565258Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7565261Z 2025-10-10T00:57:26.7565387Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7565588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7565663Z res = mod(**inputs) 2025-10-10T00:57:26.7565925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7566043Z outputs = self.model( 2025-10-10T00:57:26.7566298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7566385Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7566640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7566713Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7566945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7567025Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7567285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7567381Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7567630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7567711Z return func(*args, **kwargs) 2025-10-10T00:57:26.7567965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7568062Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7568066Z 2025-10-10T00:57:26.7568176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7568392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7568464Z res = mod(**inputs) 2025-10-10T00:57:26.7568744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7568820Z outputs = self.model( 2025-10-10T00:57:26.7569076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7569158Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7569411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7569486Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7569718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7569798Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7570062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7570161Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7570420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7570501Z return func(*args, **kwargs) 2025-10-10T00:57:26.7570770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7570872Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7570896Z 2025-10-10T00:57:26.7570988Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7571106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7571326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7571392Z res = mod(**inputs) 2025-10-10T00:57:26.7571678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7571750Z outputs = self.model( 2025-10-10T00:57:26.7572013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7572092Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7572378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7572464Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7572689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7572778Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7573032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7573126Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7573381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7573453Z return func(*args, **kwargs) 2025-10-10T00:57:26.7573720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7573834Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7574153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7574301Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7574305Z 2025-10-10T00:57:26.7574415Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7574631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7574703Z res = mod(**inputs) 2025-10-10T00:57:26.7574988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7575065Z outputs = self.model( 2025-10-10T00:57:26.7575337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7575422Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7575678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7575759Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7575984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7576073Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7576339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7576436Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7576705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7576780Z return func(*args, **kwargs) 2025-10-10T00:57:26.7577058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7577148Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7577173Z 2025-10-10T00:57:26.7577284Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7577498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7577566Z res = mod(**inputs) 2025-10-10T00:57:26.7577862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7577938Z outputs = self.model( 2025-10-10T00:57:26.7578216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7578292Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7578589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7578693Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7578931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7579024Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7579289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7579417Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7579423Z 2025-10-10T00:57:26.7579542Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7579753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7579830Z res = mod(**inputs) 2025-10-10T00:57:26.7580111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7580185Z outputs = self.model( 2025-10-10T00:57:26.7580471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7580551Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7580829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7580908Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7581168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7581257Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7581535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7581668Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7581673Z 2025-10-10T00:57:26.7581784Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7582001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7582073Z res = mod(**inputs) 2025-10-10T00:57:26.7582350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7582432Z outputs = self.model( 2025-10-10T00:57:26.7582741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7582825Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7583094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7583176Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7583420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7583506Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7583780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7583892Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7583896Z 2025-10-10T00:57:26.7584017Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7584233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7584322Z res = mod(**inputs) 2025-10-10T00:57:26.7584622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7584698Z outputs = self.model( 2025-10-10T00:57:26.7584994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7585112Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7585391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7585480Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7585728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7585826Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7586101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7586209Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7586565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7586651Z return func(*args, **kwargs) 2025-10-10T00:57:26.7586941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7587113Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7587119Z 2025-10-10T00:57:26.7587241Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7587459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7587532Z res = mod(**inputs) 2025-10-10T00:57:26.7587822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7587898Z outputs = self.model( 2025-10-10T00:57:26.7588186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7588271Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7588563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7588648Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7588892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7588989Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7589257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7589366Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7589627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7589703Z return func(*args, **kwargs) 2025-10-10T00:57:26.7589980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7590072Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7590076Z 2025-10-10T00:57:26.7590196Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7590404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7590504Z res = mod(**inputs) 2025-10-10T00:57:26.7590776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7590848Z outputs = self.model( 2025-10-10T00:57:26.7591150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7591229Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7591504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7591581Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7591852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7591948Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7592217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7592333Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7592576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7592651Z return func(*args, **kwargs) 2025-10-10T00:57:26.7592912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7593002Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7593005Z 2025-10-10T00:57:26.7593100Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7593210Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7593417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7593487Z res = mod(**inputs) 2025-10-10T00:57:26.7593761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7593842Z outputs = self.model( 2025-10-10T00:57:26.7594112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7594199Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7594465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7594542Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7594787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7594875Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7595148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7595247Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7595509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7595592Z return func(*args, **kwargs) 2025-10-10T00:57:26.7595859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7595973Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7596281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7596438Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7596443Z 2025-10-10T00:57:26.7596554Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7596768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7596869Z res = mod(**inputs) 2025-10-10T00:57:26.7597149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7597240Z outputs = self.model( 2025-10-10T00:57:26.7597526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7597606Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7597881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7597959Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7598255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7598343Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7598615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7598711Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7598970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7599054Z return func(*args, **kwargs) 2025-10-10T00:57:26.7599319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7599414Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7599418Z 2025-10-10T00:57:26.7599526Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7599740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7599820Z res = mod(**inputs) 2025-10-10T00:57:26.7600089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7600170Z outputs = self.model( 2025-10-10T00:57:26.7600445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7600529Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7600799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7600876Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7601121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7601206Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7601486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7601612Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7601618Z 2025-10-10T00:57:26.7601726Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7601944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7602014Z res = mod(**inputs) 2025-10-10T00:57:26.7602292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7602365Z outputs = self.model( 2025-10-10T00:57:26.7602636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7602721Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7602990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7603075Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7603334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7603428Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7603692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7603850Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7603854Z 2025-10-10T00:57:26.7603975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7604189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7604267Z res = mod(**inputs) 2025-10-10T00:57:26.7604556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7604649Z outputs = self.model( 2025-10-10T00:57:26.7604933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7605012Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7605295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7605372Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7605625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7605714Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7605987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7606089Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7606093Z 2025-10-10T00:57:26.7606204Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7606424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7606494Z res = mod(**inputs) 2025-10-10T00:57:26.7606769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7606849Z outputs = self.model( 2025-10-10T00:57:26.7607127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7607212Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7607485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7607563Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7607814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7607900Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7608181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7608269Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7608272Z 2025-10-10T00:57:26.7608390Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7608605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7608675Z res = mod(**inputs) 2025-10-10T00:57:26.7608960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7609033Z outputs = self.model( 2025-10-10T00:57:26.7609317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7609396Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7609667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7609776Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7610012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7610104Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7610388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7610500Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7610766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7610861Z return func(*args, **kwargs) 2025-10-10T00:57:26.7611153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7611319Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7611323Z 2025-10-10T00:57:26.7611441Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7611652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7611723Z res = mod(**inputs) 2025-10-10T00:57:26.7612008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7612081Z outputs = self.model( 2025-10-10T00:57:26.7612360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7612437Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7612717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7612793Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7613034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7613128Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7613395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7613502Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7613763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7613836Z return func(*args, **kwargs) 2025-10-10T00:57:26.7614114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7614204Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7614207Z 2025-10-10T00:57:26.7614323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7614536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7614606Z res = mod(**inputs) 2025-10-10T00:57:26.7614884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7614959Z outputs = self.model( 2025-10-10T00:57:26.7615242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7615320Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7615594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7615672Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7615910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7616021Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7616290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7616398Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7616715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7616794Z return func(*args, **kwargs) 2025-10-10T00:57:26.7617074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7617167Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7617171Z 2025-10-10T00:57:26.7617303Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7617416Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7617633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7617704Z res = mod(**inputs) 2025-10-10T00:57:26.7617979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7618060Z outputs = self.model( 2025-10-10T00:57:26.7618333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7618417Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7618683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7618761Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7619008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7619096Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7619371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7619471Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7619736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7619823Z return func(*args, **kwargs) 2025-10-10T00:57:26.7620092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7620205Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7620527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7620686Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7620690Z 2025-10-10T00:57:26.7620802Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7621015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7621093Z res = mod(**inputs) 2025-10-10T00:57:26.7621368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7621448Z outputs = self.model( 2025-10-10T00:57:26.7621723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7621802Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7622080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7622161Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7622407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7622512Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7622788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7622884Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7623168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7623256Z return func(*args, **kwargs) 2025-10-10T00:57:26.7623522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7623617Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7623621Z 2025-10-10T00:57:26.7623746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7623975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7624053Z res = mod(**inputs) 2025-10-10T00:57:26.7624338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7624417Z outputs = self.model( 2025-10-10T00:57:26.7624698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7624777Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7625052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7625130Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7625378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7625466Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7625740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7625869Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7625873Z 2025-10-10T00:57:26.7625983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7626205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7626275Z res = mod(**inputs) 2025-10-10T00:57:26.7626664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7626747Z outputs = self.model( 2025-10-10T00:57:26.7627040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7627136Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7627419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7627511Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7627768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7627878Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7628153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7628282Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7628286Z 2025-10-10T00:57:26.7628411Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7628627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7628709Z res = mod(**inputs) 2025-10-10T00:57:26.7628989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7629064Z outputs = self.model( 2025-10-10T00:57:26.7629388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7629467Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7629766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7629847Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7630106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7630195Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7630490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7630607Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7630611Z 2025-10-10T00:57:26.7630723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7630953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7631024Z res = mod(**inputs) 2025-10-10T00:57:26.7631302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7631387Z outputs = self.model( 2025-10-10T00:57:26.7631809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7631903Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7632181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7632265Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7632520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7632611Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7632895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7632998Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7633279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7633358Z return func(*args, **kwargs) 2025-10-10T00:57:26.7633633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7633811Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7633817Z 2025-10-10T00:57:26.7633931Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7634154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7634232Z res = mod(**inputs) 2025-10-10T00:57:26.7634512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7634597Z outputs = self.model( 2025-10-10T00:57:26.7634872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7634951Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7635198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7635276Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7635497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7635578Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7635833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7635980Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7636227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7636296Z return func(*args, **kwargs) 2025-10-10T00:57:26.7636564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7636658Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7636661Z 2025-10-10T00:57:26.7636763Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7637002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7637096Z res = mod(**inputs) 2025-10-10T00:57:26.7637364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7637436Z outputs = self.model( 2025-10-10T00:57:26.7637694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7637778Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7638041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7638122Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7638342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7638423Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7638677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7638767Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7639016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7639086Z return func(*args, **kwargs) 2025-10-10T00:57:26.7639337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7639433Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7639437Z 2025-10-10T00:57:26.7639523Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7639633Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7639828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7639898Z res = mod(**inputs) 2025-10-10T00:57:26.7640152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7640220Z outputs = self.model( 2025-10-10T00:57:26.7640481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7640555Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7640816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7640891Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7641116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7641204Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7641454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7641557Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7641807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7641897Z return func(*args, **kwargs) 2025-10-10T00:57:26.7642157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7642258Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7642574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7642714Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7642717Z 2025-10-10T00:57:26.7642828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7643029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7643142Z res = mod(**inputs) 2025-10-10T00:57:26.7643410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7643481Z outputs = self.model( 2025-10-10T00:57:26.7643746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7643820Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7644076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7644157Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7644381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7644468Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7644723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-10-10T00:57:26.7644823Z hidden_states, attn_weights = self.self_attn( 2025-10-10T00:57:26.7645069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7645141Z return func(*args, **kwargs) 2025-10-10T00:57:26.7645398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7645485Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7645489Z 2025-10-10T00:57:26.7645599Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7645801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7645867Z res = mod(**inputs) 2025-10-10T00:57:26.7646132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7646204Z outputs = self.model( 2025-10-10T00:57:26.7646467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7646541Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7646803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7646877Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7647103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7647191Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7647444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7647569Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7647576Z 2025-10-10T00:57:26.7647678Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7647876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7647973Z res = mod(**inputs) 2025-10-10T00:57:26.7648246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7648323Z outputs = self.model( 2025-10-10T00:57:26.7648616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7648691Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7648948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7649022Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7649270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7649375Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7649632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-10-10T00:57:26.7649753Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7649757Z 2025-10-10T00:57:26.7649861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7650070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7650137Z res = mod(**inputs) 2025-10-10T00:57:26.7650402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7650471Z outputs = self.model( 2025-10-10T00:57:26.7650729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7650812Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7651067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7651150Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7651386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7651479Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7651750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-10-10T00:57:26.7651839Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7651843Z 2025-10-10T00:57:26.7651960Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7652173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7652254Z res = mod(**inputs) 2025-10-10T00:57:26.7652529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7652604Z outputs = self.model( 2025-10-10T00:57:26.7652883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-10-10T00:57:26.7652960Z encoder_outputs = self.encoder( 2025-10-10T00:57:26.7653239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-10-10T00:57:26.7653317Z layer_outputs = encoder_layer( 2025-10-10T00:57:26.7653555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7653652Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7653918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-10-10T00:57:26.7654018Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7654021Z 2025-10-10T00:57:26.7654151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7654371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7654440Z res = mod(**inputs) 2025-10-10T00:57:26.7654732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7654813Z outputs = self.model( 2025-10-10T00:57:26.7655088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7655175Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7655463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-10-10T00:57:26.7655664Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-10-10T00:57:26.7655928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7656006Z return func(*args, **kwargs) 2025-10-10T00:57:26.7656281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7656514Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7656862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7657070Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T00:57:26.7657075Z 2025-10-10T00:57:26.7657187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7657407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7657478Z res = mod(**inputs) 2025-10-10T00:57:26.7657761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7657835Z outputs = self.model( 2025-10-10T00:57:26.7658118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7658197Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7658473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-10-10T00:57:26.7658657Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-10-10T00:57:26.7658914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 122, in decorate_context 2025-10-10T00:57:26.7659000Z return func(*args, **kwargs) 2025-10-10T00:57:26.7659269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-10-10T00:57:26.7659496Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-10-10T00:57:26.7659860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-10-10T00:57:26.7660061Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T00:57:26.7660066Z 2025-10-10T00:57:26.7660186Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7660397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7660477Z res = mod(**inputs) 2025-10-10T00:57:26.7660753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7660850Z outputs = self.model( 2025-10-10T00:57:26.7661137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7661214Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7661514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7661595Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7661834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7661930Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7662206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7662305Z return func(*args, **kwargs) 2025-10-10T00:57:26.7662572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7662691Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7662960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7663038Z return func(*args, **kwargs) 2025-10-10T00:57:26.7663313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7663473Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7663477Z 2025-10-10T00:57:26.7663595Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7663811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7663881Z res = mod(**inputs) 2025-10-10T00:57:26.7664159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7664234Z outputs = self.model( 2025-10-10T00:57:26.7664514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7664592Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7664883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7664963Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7665209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7665305Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7665567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7665651Z return func(*args, **kwargs) 2025-10-10T00:57:26.7665921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7666034Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7666391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7666483Z return func(*args, **kwargs) 2025-10-10T00:57:26.7666791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7666881Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7666885Z 2025-10-10T00:57:26.7667004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7667223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7667296Z res = mod(**inputs) 2025-10-10T00:57:26.7667578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7667680Z outputs = self.model( 2025-10-10T00:57:26.7667962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7668058Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7668331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7668419Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7668663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7668798Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7669070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7669148Z return func(*args, **kwargs) 2025-10-10T00:57:26.7669421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7669528Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7669848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7669917Z return func(*args, **kwargs) 2025-10-10T00:57:26.7670173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7670261Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7670264Z 2025-10-10T00:57:26.7670350Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7670459Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7670657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7670730Z res = mod(**inputs) 2025-10-10T00:57:26.7670981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7671048Z outputs = self.model( 2025-10-10T00:57:26.7671316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7671388Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7671637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7671707Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7671920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7672007Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7672237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7672315Z return func(*args, **kwargs) 2025-10-10T00:57:26.7672559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7672664Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7672902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7672971Z return func(*args, **kwargs) 2025-10-10T00:57:26.7673223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7673323Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7673617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7673771Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7673774Z 2025-10-10T00:57:26.7673875Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7674079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7674143Z res = mod(**inputs) 2025-10-10T00:57:26.7674421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7674489Z outputs = self.model( 2025-10-10T00:57:26.7674745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7674859Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7675108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7675190Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7675408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7675495Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7675733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7675801Z return func(*args, **kwargs) 2025-10-10T00:57:26.7676061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7676166Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7676435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7676510Z return func(*args, **kwargs) 2025-10-10T00:57:26.7676779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7676875Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7676879Z 2025-10-10T00:57:26.7676988Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7677205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7677276Z res = mod(**inputs) 2025-10-10T00:57:26.7677560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7677631Z outputs = self.model( 2025-10-10T00:57:26.7677899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7677988Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7678258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7678344Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7678579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7678664Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7678927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7679001Z return func(*args, **kwargs) 2025-10-10T00:57:26.7679273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7679392Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7679654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7679729Z return func(*args, **kwargs) 2025-10-10T00:57:26.7679995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7680189Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7680193Z 2025-10-10T00:57:26.7680303Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7680541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7680613Z res = mod(**inputs) 2025-10-10T00:57:26.7680886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7680968Z outputs = self.model( 2025-10-10T00:57:26.7681254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7681358Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7681634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7681721Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7681958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7682045Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7682317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7682392Z return func(*args, **kwargs) 2025-10-10T00:57:26.7682676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7682792Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7683041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7683123Z return func(*args, **kwargs) 2025-10-10T00:57:26.7683381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7683473Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7683476Z 2025-10-10T00:57:26.7683581Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7683784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7683859Z res = mod(**inputs) 2025-10-10T00:57:26.7684131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7684211Z outputs = self.model( 2025-10-10T00:57:26.7684486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7684570Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7684843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7684920Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7685169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7685258Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7685525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7685599Z return func(*args, **kwargs) 2025-10-10T00:57:26.7685868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7685997Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7686254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7686356Z return func(*args, **kwargs) 2025-10-10T00:57:26.7686623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7686723Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7686727Z 2025-10-10T00:57:26.7686833Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7686945Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7687165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7687234Z res = mod(**inputs) 2025-10-10T00:57:26.7687529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7687620Z outputs = self.model( 2025-10-10T00:57:26.7687888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7687977Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7688250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7688336Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7688575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7688661Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7688927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7689001Z return func(*args, **kwargs) 2025-10-10T00:57:26.7689276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7689392Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7689658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7689731Z return func(*args, **kwargs) 2025-10-10T00:57:26.7689998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7690113Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7690425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7690578Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7690582Z 2025-10-10T00:57:26.7690689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7690903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7690981Z res = mod(**inputs) 2025-10-10T00:57:26.7691256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7691334Z outputs = self.model( 2025-10-10T00:57:26.7691602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7691689Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7691959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7692035Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7692280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7692369Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7692630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7692726Z return func(*args, **kwargs) 2025-10-10T00:57:26.7692992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7693115Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7693390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7693473Z return func(*args, **kwargs) 2025-10-10T00:57:26.7693744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7693834Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7693844Z 2025-10-10T00:57:26.7693987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7694202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7694279Z res = mod(**inputs) 2025-10-10T00:57:26.7694555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7694633Z outputs = self.model( 2025-10-10T00:57:26.7694901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7694979Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7695254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7695331Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7695572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7695662Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7695920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7696002Z return func(*args, **kwargs) 2025-10-10T00:57:26.7696270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7696403Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7696407Z 2025-10-10T00:57:26.7696518Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7696737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7696806Z res = mod(**inputs) 2025-10-10T00:57:26.7697078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7697163Z outputs = self.model( 2025-10-10T00:57:26.7697433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7697520Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7697791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7697869Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7698115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7698202Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7698466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7698541Z return func(*args, **kwargs) 2025-10-10T00:57:26.7698812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7698949Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7698953Z 2025-10-10T00:57:26.7699091Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7699311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7699381Z res = mod(**inputs) 2025-10-10T00:57:26.7699711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7699786Z outputs = self.model( 2025-10-10T00:57:26.7700055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7700141Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7700424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7700528Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7700774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7700862Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7701136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7701211Z return func(*args, **kwargs) 2025-10-10T00:57:26.7701507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7701595Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7701599Z 2025-10-10T00:57:26.7701716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7701931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7702004Z res = mod(**inputs) 2025-10-10T00:57:26.7702295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7702370Z outputs = self.model( 2025-10-10T00:57:26.7702659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7702736Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7703017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7703104Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7703347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7703439Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7703711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7703789Z return func(*args, **kwargs) 2025-10-10T00:57:26.7704074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7704182Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7704453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7704527Z return func(*args, **kwargs) 2025-10-10T00:57:26.7704817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7704982Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7704986Z 2025-10-10T00:57:26.7705096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7705324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7705394Z res = mod(**inputs) 2025-10-10T00:57:26.7705683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7705776Z outputs = self.model( 2025-10-10T00:57:26.7706049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7706137Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7706519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7706614Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7706858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7706954Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7707259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7707338Z return func(*args, **kwargs) 2025-10-10T00:57:26.7707625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7707735Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7708010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7708089Z return func(*args, **kwargs) 2025-10-10T00:57:26.7708366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7708467Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7708471Z 2025-10-10T00:57:26.7708586Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7708819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7708891Z res = mod(**inputs) 2025-10-10T00:57:26.7709169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7709253Z outputs = self.model( 2025-10-10T00:57:26.7709530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7709617Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7709895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7709980Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7710222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7710315Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7710587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7710664Z return func(*args, **kwargs) 2025-10-10T00:57:26.7710949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7711058Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7711325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7711408Z return func(*args, **kwargs) 2025-10-10T00:57:26.7711684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7711787Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7711792Z 2025-10-10T00:57:26.7711881Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7712004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7712221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7712314Z res = mod(**inputs) 2025-10-10T00:57:26.7712601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7712676Z outputs = self.model( 2025-10-10T00:57:26.7712981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7713063Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7713342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7713430Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7713691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7713804Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7714069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7714149Z return func(*args, **kwargs) 2025-10-10T00:57:26.7714433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7714542Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7714819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7714897Z return func(*args, **kwargs) 2025-10-10T00:57:26.7715179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7715292Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7715612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7715772Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7715776Z 2025-10-10T00:57:26.7715889Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7716116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7716188Z res = mod(**inputs) 2025-10-10T00:57:26.7716469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7716554Z outputs = self.model( 2025-10-10T00:57:26.7716831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7716918Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7717203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7717291Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7717539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7717626Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7717899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7717979Z return func(*args, **kwargs) 2025-10-10T00:57:26.7718261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7718369Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7718631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7718711Z return func(*args, **kwargs) 2025-10-10T00:57:26.7718962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7719074Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7719077Z 2025-10-10T00:57:26.7719185Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7719399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7719500Z res = mod(**inputs) 2025-10-10T00:57:26.7719776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7719857Z outputs = self.model( 2025-10-10T00:57:26.7720128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7720248Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7720522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7720601Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7720850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7720936Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7721207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7721282Z return func(*args, **kwargs) 2025-10-10T00:57:26.7721554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-10-10T00:57:26.7721650Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7721653Z 2025-10-10T00:57:26.7721766Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7721986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7722057Z res = mod(**inputs) 2025-10-10T00:57:26.7722339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7722413Z outputs = self.model( 2025-10-10T00:57:26.7722685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7722772Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7723043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7723130Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7723369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7723456Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7723723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7723801Z return func(*args, **kwargs) 2025-10-10T00:57:26.7724076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7724193Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7724455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7724536Z return func(*args, **kwargs) 2025-10-10T00:57:26.7724805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7724971Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7724978Z 2025-10-10T00:57:26.7725089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7725307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7725406Z res = mod(**inputs) 2025-10-10T00:57:26.7725683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7725764Z outputs = self.model( 2025-10-10T00:57:26.7726058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7726144Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7726414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7726491Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7726749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7726855Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7727116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7727194Z return func(*args, **kwargs) 2025-10-10T00:57:26.7727467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7727586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7727842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7727925Z return func(*args, **kwargs) 2025-10-10T00:57:26.7728200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7728303Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7728306Z 2025-10-10T00:57:26.7728419Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7728645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7728725Z res = mod(**inputs) 2025-10-10T00:57:26.7728995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7729074Z outputs = self.model( 2025-10-10T00:57:26.7729345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7729423Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7729699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7729778Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7730022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7730103Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7730356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7730427Z return func(*args, **kwargs) 2025-10-10T00:57:26.7730679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7730799Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7731040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7731121Z return func(*args, **kwargs) 2025-10-10T00:57:26.7731390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7731707Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7731724Z 2025-10-10T00:57:26.7731820Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7731987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7732208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7732283Z res = mod(**inputs) 2025-10-10T00:57:26.7732568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7732671Z outputs = self.model( 2025-10-10T00:57:26.7732943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7733030Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7733324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7733440Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7733683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7733772Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7734039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7734115Z return func(*args, **kwargs) 2025-10-10T00:57:26.7734395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7734512Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7734771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7734854Z return func(*args, **kwargs) 2025-10-10T00:57:26.7735131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7735247Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7735569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7735724Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7735728Z 2025-10-10T00:57:26.7735840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7736056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7736134Z res = mod(**inputs) 2025-10-10T00:57:26.7736409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7736495Z outputs = self.model( 2025-10-10T00:57:26.7736771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7736852Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7737135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7737216Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7737464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7737552Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7737818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7737894Z return func(*args, **kwargs) 2025-10-10T00:57:26.7738167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7738295Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7738556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7738661Z return func(*args, **kwargs) 2025-10-10T00:57:26.7738931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7739022Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7739026Z 2025-10-10T00:57:26.7739163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7739375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7739452Z res = mod(**inputs) 2025-10-10T00:57:26.7739726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7739821Z outputs = self.model( 2025-10-10T00:57:26.7740120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7740201Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7740485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7740565Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7740812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7740901Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7741164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7741248Z return func(*args, **kwargs) 2025-10-10T00:57:26.7741520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7741661Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7741664Z 2025-10-10T00:57:26.7741777Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7741993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7742072Z res = mod(**inputs) 2025-10-10T00:57:26.7742348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7742431Z outputs = self.model( 2025-10-10T00:57:26.7742706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7742791Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7743067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7743149Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7743400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7743491Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7743766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7743843Z return func(*args, **kwargs) 2025-10-10T00:57:26.7744120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7744258Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7744262Z 2025-10-10T00:57:26.7744377Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7744600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7744675Z res = mod(**inputs) 2025-10-10T00:57:26.7744961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7745057Z outputs = self.model( 2025-10-10T00:57:26.7745346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7745434Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7745740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7745827Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7746074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7746164Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7746778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7746895Z return func(*args, **kwargs) 2025-10-10T00:57:26.7747188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7747283Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7747288Z 2025-10-10T00:57:26.7747406Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7747637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7747712Z res = mod(**inputs) 2025-10-10T00:57:26.7748020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7748094Z outputs = self.model( 2025-10-10T00:57:26.7748389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7748470Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7748758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7748847Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7749091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7749189Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7749464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7749543Z return func(*args, **kwargs) 2025-10-10T00:57:26.7749832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7749943Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7750221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7750300Z return func(*args, **kwargs) 2025-10-10T00:57:26.7750576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7750751Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7750755Z 2025-10-10T00:57:26.7750869Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7751094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7751160Z res = mod(**inputs) 2025-10-10T00:57:26.7751435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7751504Z outputs = self.model( 2025-10-10T00:57:26.7751792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7751878Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7752209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7752323Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7752546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7752628Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7752897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7752971Z return func(*args, **kwargs) 2025-10-10T00:57:26.7753232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7753335Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7753630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7753708Z return func(*args, **kwargs) 2025-10-10T00:57:26.7753976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7754074Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7754077Z 2025-10-10T00:57:26.7754189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7754410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7754482Z res = mod(**inputs) 2025-10-10T00:57:26.7754752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7754834Z outputs = self.model( 2025-10-10T00:57:26.7755115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7755202Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7755483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7755563Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7755785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7755866Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7756125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7756200Z return func(*args, **kwargs) 2025-10-10T00:57:26.7756472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7756581Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7756837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7756920Z return func(*args, **kwargs) 2025-10-10T00:57:26.7757186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7757285Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7757289Z 2025-10-10T00:57:26.7757378Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7757492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7757711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7757781Z res = mod(**inputs) 2025-10-10T00:57:26.7758066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7758138Z outputs = self.model( 2025-10-10T00:57:26.7758399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7758492Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7758750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7758830Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7759072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7759161Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7759407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7759477Z return func(*args, **kwargs) 2025-10-10T00:57:26.7759755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7759871Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7760121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7760193Z return func(*args, **kwargs) 2025-10-10T00:57:26.7760446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7760557Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7760849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7760995Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7760998Z 2025-10-10T00:57:26.7761102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7761312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7761379Z res = mod(**inputs) 2025-10-10T00:57:26.7761634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7761712Z outputs = self.model( 2025-10-10T00:57:26.7761966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7762051Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7762321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7762401Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7762646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7762733Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7763003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7763078Z return func(*args, **kwargs) 2025-10-10T00:57:26.7763353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7763458Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7763716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7763798Z return func(*args, **kwargs) 2025-10-10T00:57:26.7764063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7764160Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7764164Z 2025-10-10T00:57:26.7764274Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7764488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7764565Z res = mod(**inputs) 2025-10-10T00:57:26.7764852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7764933Z outputs = self.model( 2025-10-10T00:57:26.7765214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7765316Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7765590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7765668Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7765917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7766043Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7766309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7766386Z return func(*args, **kwargs) 2025-10-10T00:57:26.7766653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7766779Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7767036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7767117Z return func(*args, **kwargs) 2025-10-10T00:57:26.7767385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7767548Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7767560Z 2025-10-10T00:57:26.7767673Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7767886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7767966Z res = mod(**inputs) 2025-10-10T00:57:26.7768239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7768317Z outputs = self.model( 2025-10-10T00:57:26.7768587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7768665Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7768944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7769021Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7769269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7769356Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7769614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7769699Z return func(*args, **kwargs) 2025-10-10T00:57:26.7769965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7770091Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7770352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7770437Z return func(*args, **kwargs) 2025-10-10T00:57:26.7770715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7770809Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7770814Z 2025-10-10T00:57:26.7770937Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7771152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7771252Z res = mod(**inputs) 2025-10-10T00:57:26.7771536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7771611Z outputs = self.model( 2025-10-10T00:57:26.7771916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7771997Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7772287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7772365Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7772653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7772746Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7773005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7773087Z return func(*args, **kwargs) 2025-10-10T00:57:26.7773352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7773476Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7773732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7773806Z return func(*args, **kwargs) 2025-10-10T00:57:26.7774081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7774180Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7774184Z 2025-10-10T00:57:26.7774281Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7774391Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7774602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7774679Z res = mod(**inputs) 2025-10-10T00:57:26.7774954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7775038Z outputs = self.model( 2025-10-10T00:57:26.7775317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7775397Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7775686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7775769Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7776020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7776109Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7776380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7776457Z return func(*args, **kwargs) 2025-10-10T00:57:26.7776741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7776866Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7777124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7777207Z return func(*args, **kwargs) 2025-10-10T00:57:26.7777475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7777584Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7777921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7778066Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7778071Z 2025-10-10T00:57:26.7778189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7778416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7778497Z res = mod(**inputs) 2025-10-10T00:57:26.7778773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7778845Z outputs = self.model( 2025-10-10T00:57:26.7779149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7779249Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7779538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7779622Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7779868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7779968Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7780236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7780320Z return func(*args, **kwargs) 2025-10-10T00:57:26.7780598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7780728Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7780994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7781072Z return func(*args, **kwargs) 2025-10-10T00:57:26.7781361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7781453Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7781457Z 2025-10-10T00:57:26.7781584Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7781803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7781874Z res = mod(**inputs) 2025-10-10T00:57:26.7782168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7782242Z outputs = self.model( 2025-10-10T00:57:26.7782535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7782616Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7782897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7782985Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7783232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7783331Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7783599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7783684Z return func(*args, **kwargs) 2025-10-10T00:57:26.7783963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-10-10T00:57:26.7784059Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7784062Z 2025-10-10T00:57:26.7784188Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7784423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7784503Z res = mod(**inputs) 2025-10-10T00:57:26.7784781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7784855Z outputs = self.model( 2025-10-10T00:57:26.7785158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7785239Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7785526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7785643Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7785892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7785981Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7786249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7786418Z return func(*args, **kwargs) 2025-10-10T00:57:26.7786704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7786849Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7786854Z 2025-10-10T00:57:26.7786968Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7787185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7787266Z res = mod(**inputs) 2025-10-10T00:57:26.7787551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7787634Z outputs = self.model( 2025-10-10T00:57:26.7787920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7788002Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7788289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7788371Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7788624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7788712Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7788989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7789070Z return func(*args, **kwargs) 2025-10-10T00:57:26.7789347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7789493Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7789497Z 2025-10-10T00:57:26.7789613Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7789835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7789906Z res = mod(**inputs) 2025-10-10T00:57:26.7790187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7790269Z outputs = self.model( 2025-10-10T00:57:26.7790551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7790644Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7790922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7791035Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7791280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7791367Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7791665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7791744Z return func(*args, **kwargs) 2025-10-10T00:57:26.7792028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7792120Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7792124Z 2025-10-10T00:57:26.7792238Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7792504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7792578Z res = mod(**inputs) 2025-10-10T00:57:26.7792871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7792945Z outputs = self.model( 2025-10-10T00:57:26.7793227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7793315Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7793597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7793683Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7793930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7794029Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7794300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7794379Z return func(*args, **kwargs) 2025-10-10T00:57:26.7794668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7794781Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7795060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7795138Z return func(*args, **kwargs) 2025-10-10T00:57:26.7795419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7795599Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7795605Z 2025-10-10T00:57:26.7795721Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7795951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7796024Z res = mod(**inputs) 2025-10-10T00:57:26.7796317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7796393Z outputs = self.model( 2025-10-10T00:57:26.7796680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7796768Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7797051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7797141Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7797389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7797479Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7797758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7797855Z return func(*args, **kwargs) 2025-10-10T00:57:26.7798137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7798245Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7798538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7798625Z return func(*args, **kwargs) 2025-10-10T00:57:26.7798904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7799004Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7799042Z 2025-10-10T00:57:26.7799157Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7799380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7799454Z res = mod(**inputs) 2025-10-10T00:57:26.7799737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7799821Z outputs = self.model( 2025-10-10T00:57:26.7800102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7800190Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7800468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7800549Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7800801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7800891Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7801164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7801242Z return func(*args, **kwargs) 2025-10-10T00:57:26.7801527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7801639Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7801915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7802000Z return func(*args, **kwargs) 2025-10-10T00:57:26.7802267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7802371Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7802375Z 2025-10-10T00:57:26.7802464Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7802574Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7802795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7802865Z res = mod(**inputs) 2025-10-10T00:57:26.7803146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7803218Z outputs = self.model( 2025-10-10T00:57:26.7803490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7803574Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7803843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7803931Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7804167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7804296Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7804556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7804630Z return func(*args, **kwargs) 2025-10-10T00:57:26.7804928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7805037Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7805302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7805377Z return func(*args, **kwargs) 2025-10-10T00:57:26.7805664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7805795Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7806106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7806263Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7806267Z 2025-10-10T00:57:26.7806381Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7806609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7806681Z res = mod(**inputs) 2025-10-10T00:57:26.7806971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7807054Z outputs = self.model( 2025-10-10T00:57:26.7807352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7807441Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7807745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7807824Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7808082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7808169Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7808444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7808519Z return func(*args, **kwargs) 2025-10-10T00:57:26.7808834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7808950Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7809242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7809330Z return func(*args, **kwargs) 2025-10-10T00:57:26.7809618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7809714Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7809717Z 2025-10-10T00:57:26.7809828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7810046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7810127Z res = mod(**inputs) 2025-10-10T00:57:26.7810417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7810500Z outputs = self.model( 2025-10-10T00:57:26.7810798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7810880Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7811202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7811293Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7811538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7811640Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7811917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7811994Z return func(*args, **kwargs) 2025-10-10T00:57:26.7812268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7812433Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7812716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7812799Z return func(*args, **kwargs) 2025-10-10T00:57:26.7813066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7813228Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7813232Z 2025-10-10T00:57:26.7813353Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7813564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7813642Z res = mod(**inputs) 2025-10-10T00:57:26.7813918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7814003Z outputs = self.model( 2025-10-10T00:57:26.7814292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7814373Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7814660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7814739Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7814990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7815078Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7815351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7815437Z return func(*args, **kwargs) 2025-10-10T00:57:26.7815714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7815853Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7816121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7816197Z return func(*args, **kwargs) 2025-10-10T00:57:26.7816471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7816560Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7816566Z 2025-10-10T00:57:26.7816684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7816894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7816972Z res = mod(**inputs) 2025-10-10T00:57:26.7817255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7817328Z outputs = self.model( 2025-10-10T00:57:26.7817606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7817703Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7817979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7818057Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7818314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7818414Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7818684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7818771Z return func(*args, **kwargs) 2025-10-10T00:57:26.7819083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7819205Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7819487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7819561Z return func(*args, **kwargs) 2025-10-10T00:57:26.7819833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7819929Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7819932Z 2025-10-10T00:57:26.7820027Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7820139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7820350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7820426Z res = mod(**inputs) 2025-10-10T00:57:26.7820703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7820786Z outputs = self.model( 2025-10-10T00:57:26.7821066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7821146Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7821451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7821532Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7821784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7821871Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7822145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7822226Z return func(*args, **kwargs) 2025-10-10T00:57:26.7822500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7822631Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7822896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7822978Z return func(*args, **kwargs) 2025-10-10T00:57:26.7823259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7823371Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7823701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7823852Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7823857Z 2025-10-10T00:57:26.7823977Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7824193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7824304Z res = mod(**inputs) 2025-10-10T00:57:26.7824585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7824661Z outputs = self.model( 2025-10-10T00:57:26.7824975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7825058Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7825345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7825425Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7825707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7825806Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7826073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7826159Z return func(*args, **kwargs) 2025-10-10T00:57:26.7826526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7826657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7826931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7827008Z return func(*args, **kwargs) 2025-10-10T00:57:26.7827296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7827395Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7827400Z 2025-10-10T00:57:26.7827523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7827742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7827813Z res = mod(**inputs) 2025-10-10T00:57:26.7828104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7828181Z outputs = self.model( 2025-10-10T00:57:26.7828470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7828551Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7828831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7828920Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7829171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7829267Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7829534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7829618Z return func(*args, **kwargs) 2025-10-10T00:57:26.7829901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7830038Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7830045Z 2025-10-10T00:57:26.7830169Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7830386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7830464Z res = mod(**inputs) 2025-10-10T00:57:26.7830748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7830823Z outputs = self.model( 2025-10-10T00:57:26.7831110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7831218Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7831654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7831799Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7832049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7832146Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7832416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7832554Z return func(*args, **kwargs) 2025-10-10T00:57:26.7832832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7832973Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7832977Z 2025-10-10T00:57:26.7833092Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7833311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7833394Z res = mod(**inputs) 2025-10-10T00:57:26.7833674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7833758Z outputs = self.model( 2025-10-10T00:57:26.7834038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7834118Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7834408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7834490Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7834744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7834834Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7835105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7835183Z return func(*args, **kwargs) 2025-10-10T00:57:26.7835460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7835559Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7835564Z 2025-10-10T00:57:26.7835679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7835905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7835977Z res = mod(**inputs) 2025-10-10T00:57:26.7836256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7836340Z outputs = self.model( 2025-10-10T00:57:26.7836615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7836704Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7836993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7837070Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7837313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7837402Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7837666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7837769Z return func(*args, **kwargs) 2025-10-10T00:57:26.7838042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-10-10T00:57:26.7838130Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7838133Z 2025-10-10T00:57:26.7838243Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7838482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7838551Z res = mod(**inputs) 2025-10-10T00:57:26.7838830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7838903Z outputs = self.model( 2025-10-10T00:57:26.7839210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7839295Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7839569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7839654Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7839894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7839981Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7840258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7840335Z return func(*args, **kwargs) 2025-10-10T00:57:26.7840614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7840727Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7840998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7841077Z return func(*args, **kwargs) 2025-10-10T00:57:26.7841351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7841526Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7841530Z 2025-10-10T00:57:26.7841644Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7841867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7841939Z res = mod(**inputs) 2025-10-10T00:57:26.7842214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7842300Z outputs = self.model( 2025-10-10T00:57:26.7842575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7842664Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7842936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7843024Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7843262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7843350Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7843620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7843694Z return func(*args, **kwargs) 2025-10-10T00:57:26.7843977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7844085Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7844347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7844457Z return func(*args, **kwargs) 2025-10-10T00:57:26.7844724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7844822Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7844841Z 2025-10-10T00:57:26.7844953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7845171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7845242Z res = mod(**inputs) 2025-10-10T00:57:26.7845531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7845638Z outputs = self.model( 2025-10-10T00:57:26.7845906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7845992Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7846262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7846342Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7846587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7846673Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7846939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7847015Z return func(*args, **kwargs) 2025-10-10T00:57:26.7847286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7847402Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7847663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7847747Z return func(*args, **kwargs) 2025-10-10T00:57:26.7848016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7848118Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7848122Z 2025-10-10T00:57:26.7848210Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7848322Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7848541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7848612Z res = mod(**inputs) 2025-10-10T00:57:26.7848895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7848969Z outputs = self.model( 2025-10-10T00:57:26.7849243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7849329Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7849664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7849751Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7849988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7850074Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7850351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7850428Z return func(*args, **kwargs) 2025-10-10T00:57:26.7850703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7850830Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7851106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7851180Z return func(*args, **kwargs) 2025-10-10T00:57:26.7851467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7851604Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7851914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7852068Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7852104Z 2025-10-10T00:57:26.7852214Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7852425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7852508Z res = mod(**inputs) 2025-10-10T00:57:26.7852783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7852861Z outputs = self.model( 2025-10-10T00:57:26.7853141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7853226Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7853509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7853585Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7853833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7853919Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7854195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7854270Z return func(*args, **kwargs) 2025-10-10T00:57:26.7854538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7854652Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7854911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7854991Z return func(*args, **kwargs) 2025-10-10T00:57:26.7855262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7855362Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7855366Z 2025-10-10T00:57:26.7855475Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7855689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7855769Z res = mod(**inputs) 2025-10-10T00:57:26.7856043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7856122Z outputs = self.model( 2025-10-10T00:57:26.7856397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7856476Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7856765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7856844Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7857095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7857181Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7857474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7857556Z return func(*args, **kwargs) 2025-10-10T00:57:26.7857823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7857964Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7858226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7858308Z return func(*args, **kwargs) 2025-10-10T00:57:26.7858575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7858775Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7858780Z 2025-10-10T00:57:26.7858900Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7859117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7859194Z res = mod(**inputs) 2025-10-10T00:57:26.7859467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7859542Z outputs = self.model( 2025-10-10T00:57:26.7859822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7859901Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7860182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7860263Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7860509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7860597Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7860861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7860948Z return func(*args, **kwargs) 2025-10-10T00:57:26.7861226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7861355Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7861623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7861700Z return func(*args, **kwargs) 2025-10-10T00:57:26.7861989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7862082Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7862086Z 2025-10-10T00:57:26.7862216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7862430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7862500Z res = mod(**inputs) 2025-10-10T00:57:26.7862782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7862855Z outputs = self.model( 2025-10-10T00:57:26.7863134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7863212Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7863500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7863584Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7863831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7863950Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7864214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7864300Z return func(*args, **kwargs) 2025-10-10T00:57:26.7864592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7864715Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7864989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7865065Z return func(*args, **kwargs) 2025-10-10T00:57:26.7865379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7865477Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7865483Z 2025-10-10T00:57:26.7865582Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7865696Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7865915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7865996Z res = mod(**inputs) 2025-10-10T00:57:26.7866277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7866435Z outputs = self.model( 2025-10-10T00:57:26.7866720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7866805Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7867101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7867187Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7867458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7867548Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7867812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7867901Z return func(*args, **kwargs) 2025-10-10T00:57:26.7868175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7868304Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7868568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7868664Z return func(*args, **kwargs) 2025-10-10T00:57:26.7868922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7869030Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7869338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7869480Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7869485Z 2025-10-10T00:57:26.7869601Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7869804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7869875Z res = mod(**inputs) 2025-10-10T00:57:26.7870146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7870221Z outputs = self.model( 2025-10-10T00:57:26.7870487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7870581Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7870843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7870918Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7871159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7871251Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7871495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7871572Z return func(*args, **kwargs) 2025-10-10T00:57:26.7871872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7871983Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7872238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7872313Z return func(*args, **kwargs) 2025-10-10T00:57:26.7872587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7872681Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7872685Z 2025-10-10T00:57:26.7872806Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7873018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7873087Z res = mod(**inputs) 2025-10-10T00:57:26.7873368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7873442Z outputs = self.model( 2025-10-10T00:57:26.7873721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7873801Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7874070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7874155Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7874392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7874486Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7874742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7874816Z return func(*args, **kwargs) 2025-10-10T00:57:26.7875096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7875226Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7875231Z 2025-10-10T00:57:26.7875350Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7875562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7875640Z res = mod(**inputs) 2025-10-10T00:57:26.7875911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7875984Z outputs = self.model( 2025-10-10T00:57:26.7876259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7876335Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7876616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7876693Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7876952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7877045Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7877302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7877399Z return func(*args, **kwargs) 2025-10-10T00:57:26.7877668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7877794Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7877805Z 2025-10-10T00:57:26.7877914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7878167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7878247Z res = mod(**inputs) 2025-10-10T00:57:26.7878521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7878605Z outputs = self.model( 2025-10-10T00:57:26.7878879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7878959Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7879245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7879326Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7879575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7879664Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7879931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7880016Z return func(*args, **kwargs) 2025-10-10T00:57:26.7880289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7880388Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7880392Z 2025-10-10T00:57:26.7880505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7880727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7880800Z res = mod(**inputs) 2025-10-10T00:57:26.7881075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7881161Z outputs = self.model( 2025-10-10T00:57:26.7881434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7881525Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7881798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7881880Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7882126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7882216Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7882487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7882563Z return func(*args, **kwargs) 2025-10-10T00:57:26.7882837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7882960Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7883219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7883321Z return func(*args, **kwargs) 2025-10-10T00:57:26.7883588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7883759Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7883763Z 2025-10-10T00:57:26.7883890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7884103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7884179Z res = mod(**inputs) 2025-10-10T00:57:26.7884451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7884565Z outputs = self.model( 2025-10-10T00:57:26.7884843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7884924Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7885212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7885291Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7885550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7885634Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7885897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7885970Z return func(*args, **kwargs) 2025-10-10T00:57:26.7886239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7886353Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7886612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7886696Z return func(*args, **kwargs) 2025-10-10T00:57:26.7886974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7887065Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7887068Z 2025-10-10T00:57:26.7887189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7887415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7887490Z res = mod(**inputs) 2025-10-10T00:57:26.7887763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7887839Z outputs = self.model( 2025-10-10T00:57:26.7888118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7888196Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7888479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7888557Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7888804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7888890Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7889153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7889235Z return func(*args, **kwargs) 2025-10-10T00:57:26.7889508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7889622Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7889883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7889979Z return func(*args, **kwargs) 2025-10-10T00:57:26.7890255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7890366Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7890371Z 2025-10-10T00:57:26.7890471Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7890580Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7890799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7890869Z res = mod(**inputs) 2025-10-10T00:57:26.7891173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7891255Z outputs = self.model( 2025-10-10T00:57:26.7891523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7891609Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7891878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7891960Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7892205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7892290Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7892553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7892631Z return func(*args, **kwargs) 2025-10-10T00:57:26.7892897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7893022Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7893277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7893359Z return func(*args, **kwargs) 2025-10-10T00:57:26.7893628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7893741Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7894051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7894194Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7894200Z 2025-10-10T00:57:26.7894316Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7894532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7894612Z res = mod(**inputs) 2025-10-10T00:57:26.7894881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7894956Z outputs = self.model( 2025-10-10T00:57:26.7895236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7895316Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7895595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7895675Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7895927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7896017Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7896285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7896390Z return func(*args, **kwargs) 2025-10-10T00:57:26.7896655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7896766Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7897046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7897123Z return func(*args, **kwargs) 2025-10-10T00:57:26.7897398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7897532Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7897552Z 2025-10-10T00:57:26.7897670Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7897884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7897962Z res = mod(**inputs) 2025-10-10T00:57:26.7898240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7898313Z outputs = self.model( 2025-10-10T00:57:26.7898596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7898675Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7898953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7899032Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7899274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7899366Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7899629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7899709Z return func(*args, **kwargs) 2025-10-10T00:57:26.7899977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-10-10T00:57:26.7900069Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7900080Z 2025-10-10T00:57:26.7900190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7900403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7900481Z res = mod(**inputs) 2025-10-10T00:57:26.7900758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7900841Z outputs = self.model( 2025-10-10T00:57:26.7901114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7901195Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7901474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7901550Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7901797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7901883Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7902143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7902227Z return func(*args, **kwargs) 2025-10-10T00:57:26.7902508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7902636Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7902921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7903003Z return func(*args, **kwargs) 2025-10-10T00:57:26.7903280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7903466Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7903470Z 2025-10-10T00:57:26.7903594Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7903810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7903889Z res = mod(**inputs) 2025-10-10T00:57:26.7904233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7904310Z outputs = self.model( 2025-10-10T00:57:26.7904604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7904686Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7904973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7905056Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7905300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7905398Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7905669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7905758Z return func(*args, **kwargs) 2025-10-10T00:57:26.7906035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7906165Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7906522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7906605Z return func(*args, **kwargs) 2025-10-10T00:57:26.7906896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7906988Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7906992Z 2025-10-10T00:57:26.7907114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7907334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7907411Z res = mod(**inputs) 2025-10-10T00:57:26.7907701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7907781Z outputs = self.model( 2025-10-10T00:57:26.7908067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7908148Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7908439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7908522Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7908770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7908871Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7909140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7909229Z return func(*args, **kwargs) 2025-10-10T00:57:26.7909510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7909657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7909927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7910003Z return func(*args, **kwargs) 2025-10-10T00:57:26.7910305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7910402Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7910406Z 2025-10-10T00:57:26.7910504Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7910619Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7910871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7910952Z res = mod(**inputs) 2025-10-10T00:57:26.7911233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7911319Z outputs = self.model( 2025-10-10T00:57:26.7911597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7911677Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7911967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7912049Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7912301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7912390Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7912661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7912746Z return func(*args, **kwargs) 2025-10-10T00:57:26.7913025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7913153Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7913420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7913502Z return func(*args, **kwargs) 2025-10-10T00:57:26.7913780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7913889Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7914223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7914376Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7914379Z 2025-10-10T00:57:26.7914504Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7914723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7914794Z res = mod(**inputs) 2025-10-10T00:57:26.7915089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7915163Z outputs = self.model( 2025-10-10T00:57:26.7915452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7915532Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7915817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7915906Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7916151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7916268Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7916534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7946502Z return func(*args, **kwargs) 2025-10-10T00:57:26.7947260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7947407Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7947733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7947819Z return func(*args, **kwargs) 2025-10-10T00:57:26.7948202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7948302Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7948315Z 2025-10-10T00:57:26.7948445Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7948667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7948742Z res = mod(**inputs) 2025-10-10T00:57:26.7949031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7949113Z outputs = self.model( 2025-10-10T00:57:26.7949388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7949471Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7949737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7949829Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7950065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7950161Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7950417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7950495Z return func(*args, **kwargs) 2025-10-10T00:57:26.7950771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7950903Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7950908Z 2025-10-10T00:57:26.7951031Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7951249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7951330Z res = mod(**inputs) 2025-10-10T00:57:26.7951597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7951675Z outputs = self.model( 2025-10-10T00:57:26.7951943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7952023Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7952293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7952371Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7952601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7952697Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7952950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7953033Z return func(*args, **kwargs) 2025-10-10T00:57:26.7953345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7953473Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7953477Z 2025-10-10T00:57:26.7953582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7953803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7953879Z res = mod(**inputs) 2025-10-10T00:57:26.7954134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7954216Z outputs = self.model( 2025-10-10T00:57:26.7954488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7954579Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7954841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7954915Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7955137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7955220Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7955461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7955541Z return func(*args, **kwargs) 2025-10-10T00:57:26.7955787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.7955882Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.7955887Z 2025-10-10T00:57:26.7955993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7956198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7956266Z res = mod(**inputs) 2025-10-10T00:57:26.7956519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7956595Z outputs = self.model( 2025-10-10T00:57:26.7956848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7956928Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7957177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7957250Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7957480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7957560Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7957807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7957879Z return func(*args, **kwargs) 2025-10-10T00:57:26.7958134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7958238Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7958478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7958559Z return func(*args, **kwargs) 2025-10-10T00:57:26.7958806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7958972Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7958976Z 2025-10-10T00:57:26.7959082Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7959297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7959370Z res = mod(**inputs) 2025-10-10T00:57:26.7959624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7959701Z outputs = self.model( 2025-10-10T00:57:26.7960012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7960096Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7960356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7960465Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7960700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7960784Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7961041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7961114Z return func(*args, **kwargs) 2025-10-10T00:57:26.7961368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7961481Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7961726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7961806Z return func(*args, **kwargs) 2025-10-10T00:57:26.7962068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7962151Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7962161Z 2025-10-10T00:57:26.7962264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7962462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7962536Z res = mod(**inputs) 2025-10-10T00:57:26.7962787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7962867Z outputs = self.model( 2025-10-10T00:57:26.7963167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7963247Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7963545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7963630Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7963879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7963962Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7964208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7964288Z return func(*args, **kwargs) 2025-10-10T00:57:26.7964553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7964671Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7964940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7965016Z return func(*args, **kwargs) 2025-10-10T00:57:26.7965289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7965383Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7965410Z 2025-10-10T00:57:26.7965507Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7965615Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7965827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7965893Z res = mod(**inputs) 2025-10-10T00:57:26.7966173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7966253Z outputs = self.model( 2025-10-10T00:57:26.7966530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7966616Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7966946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7967034Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7967270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7967353Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7967606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7967681Z return func(*args, **kwargs) 2025-10-10T00:57:26.7967943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7968046Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7968292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7968380Z return func(*args, **kwargs) 2025-10-10T00:57:26.7968635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7968748Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7969058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7969211Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7969215Z 2025-10-10T00:57:26.7969338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7969554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7969631Z res = mod(**inputs) 2025-10-10T00:57:26.7969904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7969987Z outputs = self.model( 2025-10-10T00:57:26.7970278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7970361Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7970715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7970793Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7971045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7971131Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7971396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7971482Z return func(*args, **kwargs) 2025-10-10T00:57:26.7971763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.7971878Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.7972136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7972232Z return func(*args, **kwargs) 2025-10-10T00:57:26.7972522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7972614Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7972636Z 2025-10-10T00:57:26.7972759Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7972973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7973051Z res = mod(**inputs) 2025-10-10T00:57:26.7973345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7973435Z outputs = self.model( 2025-10-10T00:57:26.7973721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7973802Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7974093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7974173Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7974453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7974551Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7974822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7974907Z return func(*args, **kwargs) 2025-10-10T00:57:26.7975180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7975300Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7975572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7975648Z return func(*args, **kwargs) 2025-10-10T00:57:26.7975927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.7976094Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.7976098Z 2025-10-10T00:57:26.7976219Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7976436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7976506Z res = mod(**inputs) 2025-10-10T00:57:26.7976804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7976879Z outputs = self.model( 2025-10-10T00:57:26.7977171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7977249Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7977530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7977620Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7977862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7977960Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7978230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7978318Z return func(*args, **kwargs) 2025-10-10T00:57:26.7978588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7978725Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7979006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7979081Z return func(*args, **kwargs) 2025-10-10T00:57:26.7979375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.7979467Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.7979471Z 2025-10-10T00:57:26.7979585Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7979805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7979897Z res = mod(**inputs) 2025-10-10T00:57:26.7980203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7980278Z outputs = self.model( 2025-10-10T00:57:26.7980578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7980658Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7980942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7981031Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7981267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7981361Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7981623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7981703Z return func(*args, **kwargs) 2025-10-10T00:57:26.7981992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7982114Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7982390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7982466Z return func(*args, **kwargs) 2025-10-10T00:57:26.7982744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.7982854Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.7982858Z 2025-10-10T00:57:26.7982951Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.7983078Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7983302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7983386Z res = mod(**inputs) 2025-10-10T00:57:26.7983666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7983745Z outputs = self.model( 2025-10-10T00:57:26.7984037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7984119Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7984411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7984491Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7984734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7984833Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7985102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7985186Z return func(*args, **kwargs) 2025-10-10T00:57:26.7985486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7985603Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7985880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7985975Z return func(*args, **kwargs) 2025-10-10T00:57:26.7986266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.7986478Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.7986852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.7987024Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.7987028Z 2025-10-10T00:57:26.7987147Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7987384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7987456Z res = mod(**inputs) 2025-10-10T00:57:26.7987740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7987815Z outputs = self.model( 2025-10-10T00:57:26.7988088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7988175Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7988453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7988543Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7988789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7988890Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7989156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7989234Z return func(*args, **kwargs) 2025-10-10T00:57:26.7989522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.7989642Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.7989917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7989993Z return func(*args, **kwargs) 2025-10-10T00:57:26.7990277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.7990380Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.7990386Z 2025-10-10T00:57:26.7990503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7990734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7990808Z res = mod(**inputs) 2025-10-10T00:57:26.7991100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7991179Z outputs = self.model( 2025-10-10T00:57:26.7991459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7991553Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7991859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7991950Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7992197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7992310Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7992594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7992671Z return func(*args, **kwargs) 2025-10-10T00:57:26.7992998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-10-10T00:57:26.7993091Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.7993095Z 2025-10-10T00:57:26.7993211Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7993438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7993547Z res = mod(**inputs) 2025-10-10T00:57:26.7993833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7993912Z outputs = self.model( 2025-10-10T00:57:26.7994200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7994281Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7994561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7994653Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7994896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7994992Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7995258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7995338Z return func(*args, **kwargs) 2025-10-10T00:57:26.7995624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7995763Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7995767Z 2025-10-10T00:57:26.7995902Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7996117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7996195Z res = mod(**inputs) 2025-10-10T00:57:26.7996468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7996542Z outputs = self.model( 2025-10-10T00:57:26.7996830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7996905Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7997168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7997246Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.7997469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.7997558Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.7997803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.7997884Z return func(*args, **kwargs) 2025-10-10T00:57:26.7998136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.7998259Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.7998273Z 2025-10-10T00:57:26.7998379Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.7998583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.7998680Z res = mod(**inputs) 2025-10-10T00:57:26.7998955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.7999038Z outputs = self.model( 2025-10-10T00:57:26.7999329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.7999408Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.7999687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.7999767Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8000029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8000137Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8000393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8000479Z return func(*args, **kwargs) 2025-10-10T00:57:26.8000747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8000847Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8000853Z 2025-10-10T00:57:26.8000964Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8001184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8001258Z res = mod(**inputs) 2025-10-10T00:57:26.8001532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8001618Z outputs = self.model( 2025-10-10T00:57:26.8001888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8001975Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8002243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8002323Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8002568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8002654Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8002922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8002996Z return func(*args, **kwargs) 2025-10-10T00:57:26.8003268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8003384Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8003644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8003728Z return func(*args, **kwargs) 2025-10-10T00:57:26.8003997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8004168Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8004173Z 2025-10-10T00:57:26.8004283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8004497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8004577Z res = mod(**inputs) 2025-10-10T00:57:26.8004851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8004932Z outputs = self.model( 2025-10-10T00:57:26.8005202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8005303Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8005584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8005693Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8005938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8006026Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8006293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8006401Z return func(*args, **kwargs) 2025-10-10T00:57:26.8006678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8006798Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8007062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8007146Z return func(*args, **kwargs) 2025-10-10T00:57:26.8007422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8007514Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8007518Z 2025-10-10T00:57:26.8007640Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8007855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8007936Z res = mod(**inputs) 2025-10-10T00:57:26.8008215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8008298Z outputs = self.model( 2025-10-10T00:57:26.8008583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8008668Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8008945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8009023Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8009272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8009359Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8009626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8009705Z return func(*args, **kwargs) 2025-10-10T00:57:26.8009977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8010093Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8010354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8010436Z return func(*args, **kwargs) 2025-10-10T00:57:26.8010716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8010820Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8010824Z 2025-10-10T00:57:26.8010915Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8011029Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8011269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8011340Z res = mod(**inputs) 2025-10-10T00:57:26.8011623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8011715Z outputs = self.model( 2025-10-10T00:57:26.8011984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8012070Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8012357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8012443Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8012680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8012765Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8013062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8013138Z return func(*args, **kwargs) 2025-10-10T00:57:26.8013412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8013520Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8013787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8013862Z return func(*args, **kwargs) 2025-10-10T00:57:26.8014129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8014246Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8014561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8014717Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8014721Z 2025-10-10T00:57:26.8014831Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8015057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8015131Z res = mod(**inputs) 2025-10-10T00:57:26.8015407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8015491Z outputs = self.model( 2025-10-10T00:57:26.8015762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8015846Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8016132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8016216Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8016467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8016558Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8016842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8016918Z return func(*args, **kwargs) 2025-10-10T00:57:26.8017195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8017312Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8017588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8017672Z return func(*args, **kwargs) 2025-10-10T00:57:26.8017950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8018052Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8018056Z 2025-10-10T00:57:26.8018203Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8018422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8018515Z res = mod(**inputs) 2025-10-10T00:57:26.8018813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8018894Z outputs = self.model( 2025-10-10T00:57:26.8019177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8019255Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8019559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8019657Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8019902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8019989Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8020260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8020340Z return func(*args, **kwargs) 2025-10-10T00:57:26.8020612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8020737Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8021008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8021089Z return func(*args, **kwargs) 2025-10-10T00:57:26.8021362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8021525Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8021531Z 2025-10-10T00:57:26.8021655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8021876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8021956Z res = mod(**inputs) 2025-10-10T00:57:26.8022250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8022326Z outputs = self.model( 2025-10-10T00:57:26.8022630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8022713Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8023011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8023093Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8023350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8023435Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8023719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8023800Z return func(*args, **kwargs) 2025-10-10T00:57:26.8024070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8024190Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8024457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8024534Z return func(*args, **kwargs) 2025-10-10T00:57:26.8024820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8024931Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8024935Z 2025-10-10T00:57:26.8025055Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8025273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8025351Z res = mod(**inputs) 2025-10-10T00:57:26.8025653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8025730Z outputs = self.model( 2025-10-10T00:57:26.8026016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8026094Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8026501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8026592Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8026837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8026933Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8027201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8027289Z return func(*args, **kwargs) 2025-10-10T00:57:26.8027570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8027689Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8027976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8028053Z return func(*args, **kwargs) 2025-10-10T00:57:26.8028334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8028431Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8028435Z 2025-10-10T00:57:26.8028530Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8028643Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8028859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8028935Z res = mod(**inputs) 2025-10-10T00:57:26.8029214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8029295Z outputs = self.model( 2025-10-10T00:57:26.8029567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8029647Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8029930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8030010Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8030257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8030343Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8030606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8030689Z return func(*args, **kwargs) 2025-10-10T00:57:26.8030959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8031083Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8031346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8031429Z return func(*args, **kwargs) 2025-10-10T00:57:26.8031880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8031988Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8032378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8032528Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8032532Z 2025-10-10T00:57:26.8032654Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8032871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8032971Z res = mod(**inputs) 2025-10-10T00:57:26.8033276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8033352Z outputs = self.model( 2025-10-10T00:57:26.8033636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8033715Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8033997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8034076Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8034319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8034408Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8034646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8034727Z return func(*args, **kwargs) 2025-10-10T00:57:26.8034979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8035091Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8035342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8035412Z return func(*args, **kwargs) 2025-10-10T00:57:26.8035678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8035765Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8035768Z 2025-10-10T00:57:26.8035877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8036071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8036139Z res = mod(**inputs) 2025-10-10T00:57:26.8036396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8036467Z outputs = self.model( 2025-10-10T00:57:26.8036722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8036794Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8037051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8037133Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8037355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8037444Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8037689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8037760Z return func(*args, **kwargs) 2025-10-10T00:57:26.8038018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8038164Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8038168Z 2025-10-10T00:57:26.8038279Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8038497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8038569Z res = mod(**inputs) 2025-10-10T00:57:26.8038831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8038898Z outputs = self.model( 2025-10-10T00:57:26.8039166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8039262Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8039515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8039590Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8039809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8039899Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8040139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8040218Z return func(*args, **kwargs) 2025-10-10T00:57:26.8040469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8040589Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8040601Z 2025-10-10T00:57:26.8040710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8040914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8040991Z res = mod(**inputs) 2025-10-10T00:57:26.8041249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8041328Z outputs = self.model( 2025-10-10T00:57:26.8041588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8041664Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8041929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8042005Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8042240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8042325Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8042579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8042659Z return func(*args, **kwargs) 2025-10-10T00:57:26.8042908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8043000Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8043006Z 2025-10-10T00:57:26.8043109Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8043311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8043377Z res = mod(**inputs) 2025-10-10T00:57:26.8043630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8043711Z outputs = self.model( 2025-10-10T00:57:26.8043963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8044060Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8044307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8044378Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8044616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8044697Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8044943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8045011Z return func(*args, **kwargs) 2025-10-10T00:57:26.8045275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-10-10T00:57:26.8045377Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.8045381Z 2025-10-10T00:57:26.8045483Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8045692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8045758Z res = mod(**inputs) 2025-10-10T00:57:26.8046030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8046100Z outputs = self.model( 2025-10-10T00:57:26.8046359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8046439Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8046699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8046792Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8047011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8047091Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8047337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8047405Z return func(*args, **kwargs) 2025-10-10T00:57:26.8047663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8047764Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8048010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8048078Z return func(*args, **kwargs) 2025-10-10T00:57:26.8048329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8048487Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8048493Z 2025-10-10T00:57:26.8048605Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8048810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8048875Z res = mod(**inputs) 2025-10-10T00:57:26.8049138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8049220Z outputs = self.model( 2025-10-10T00:57:26.8049494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8049577Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8049852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8049928Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8050200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8050285Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8050538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8050633Z return func(*args, **kwargs) 2025-10-10T00:57:26.8050878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8050987Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8051224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8051330Z return func(*args, **kwargs) 2025-10-10T00:57:26.8051580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8051664Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8051668Z 2025-10-10T00:57:26.8051781Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8051981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8052056Z res = mod(**inputs) 2025-10-10T00:57:26.8052311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8052380Z outputs = self.model( 2025-10-10T00:57:26.8052640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8052713Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8052977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8053052Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8053283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8053364Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8053606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8053687Z return func(*args, **kwargs) 2025-10-10T00:57:26.8053939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8054046Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8054291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8054364Z return func(*args, **kwargs) 2025-10-10T00:57:26.8054624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8054716Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8054719Z 2025-10-10T00:57:26.8054810Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8054914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8055114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8055189Z res = mod(**inputs) 2025-10-10T00:57:26.8055449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8055526Z outputs = self.model( 2025-10-10T00:57:26.8055782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8055862Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8056117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8056209Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8056438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8056518Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8056790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8056866Z return func(*args, **kwargs) 2025-10-10T00:57:26.8057139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8057252Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8057540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8057620Z return func(*args, **kwargs) 2025-10-10T00:57:26.8057874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8057973Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8058275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8058412Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8058416Z 2025-10-10T00:57:26.8058525Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8058725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8058796Z res = mod(**inputs) 2025-10-10T00:57:26.8059056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8059127Z outputs = self.model( 2025-10-10T00:57:26.8059393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8059467Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8059730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8059805Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8060042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8060137Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8060393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8060481Z return func(*args, **kwargs) 2025-10-10T00:57:26.8060744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8060852Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8061098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8061168Z return func(*args, **kwargs) 2025-10-10T00:57:26.8061431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8061517Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8061520Z 2025-10-10T00:57:26.8061634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8061853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8061929Z res = mod(**inputs) 2025-10-10T00:57:26.8062220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8062316Z outputs = self.model( 2025-10-10T00:57:26.8062603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8062684Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8063001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8063084Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8063326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8063420Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8063709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8063810Z return func(*args, **kwargs) 2025-10-10T00:57:26.8064092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8064217Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8064496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8064572Z return func(*args, **kwargs) 2025-10-10T00:57:26.8064867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8065035Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8065040Z 2025-10-10T00:57:26.8065162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8065383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8065456Z res = mod(**inputs) 2025-10-10T00:57:26.8065747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8065824Z outputs = self.model( 2025-10-10T00:57:26.8066112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8066191Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8066548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8066645Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8066890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8066988Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8067260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8067338Z return func(*args, **kwargs) 2025-10-10T00:57:26.8067631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8067750Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8068017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8068096Z return func(*args, **kwargs) 2025-10-10T00:57:26.8068382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8068476Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8068480Z 2025-10-10T00:57:26.8068596Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8068832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8068903Z res = mod(**inputs) 2025-10-10T00:57:26.8069187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8069283Z outputs = self.model( 2025-10-10T00:57:26.8069551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8069638Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8069926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8070018Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8070256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8070359Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8070644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8070719Z return func(*args, **kwargs) 2025-10-10T00:57:26.8070998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8071112Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8071381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8071456Z return func(*args, **kwargs) 2025-10-10T00:57:26.8071725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8071825Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8071828Z 2025-10-10T00:57:26.8071916Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8072037Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8072251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8072322Z res = mod(**inputs) 2025-10-10T00:57:26.8072612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8072688Z outputs = self.model( 2025-10-10T00:57:26.8072977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8073056Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8073345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8073424Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8073670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8073768Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8074036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8074121Z return func(*args, **kwargs) 2025-10-10T00:57:26.8074399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8074516Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8074797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8074870Z return func(*args, **kwargs) 2025-10-10T00:57:26.8075148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8075259Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8075572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8075745Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8075749Z 2025-10-10T00:57:26.8075864Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8076092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8076164Z res = mod(**inputs) 2025-10-10T00:57:26.8076468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8076545Z outputs = self.model( 2025-10-10T00:57:26.8076824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8076912Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8077227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8077316Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8077563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8077652Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8077933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8078014Z return func(*args, **kwargs) 2025-10-10T00:57:26.8078295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8078411Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8078678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8078756Z return func(*args, **kwargs) 2025-10-10T00:57:26.8079024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8079125Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8079129Z 2025-10-10T00:57:26.8079240Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8079459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8079530Z res = mod(**inputs) 2025-10-10T00:57:26.8079801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8079882Z outputs = self.model( 2025-10-10T00:57:26.8080152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8080243Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8080512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8080591Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8080834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8080920Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8081188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8081263Z return func(*args, **kwargs) 2025-10-10T00:57:26.8081538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8081669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8081673Z 2025-10-10T00:57:26.8081787Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8082006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8082096Z res = mod(**inputs) 2025-10-10T00:57:26.8082376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8082449Z outputs = self.model( 2025-10-10T00:57:26.8082734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8082821Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8083089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8083174Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8083428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8083540Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8083803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8083881Z return func(*args, **kwargs) 2025-10-10T00:57:26.8084161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8084292Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8084296Z 2025-10-10T00:57:26.8084423Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8084642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8084714Z res = mod(**inputs) 2025-10-10T00:57:26.8085007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8085084Z outputs = self.model( 2025-10-10T00:57:26.8085362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8085442Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8085720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8085797Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8086037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8086133Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8086401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8086486Z return func(*args, **kwargs) 2025-10-10T00:57:26.8086762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8086856Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8086860Z 2025-10-10T00:57:26.8086987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8087208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8087286Z res = mod(**inputs) 2025-10-10T00:57:26.8087566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8087642Z outputs = self.model( 2025-10-10T00:57:26.8087929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8088008Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8088293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8088373Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8088625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8088731Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8088995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8089078Z return func(*args, **kwargs) 2025-10-10T00:57:26.8089384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8089505Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8089768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8089844Z return func(*args, **kwargs) 2025-10-10T00:57:26.8090166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8090335Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8090341Z 2025-10-10T00:57:26.8090463Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8090680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8090758Z res = mod(**inputs) 2025-10-10T00:57:26.8091047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8091124Z outputs = self.model( 2025-10-10T00:57:26.8091411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8091492Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8091816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8091899Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8092151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8092251Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8092527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8092612Z return func(*args, **kwargs) 2025-10-10T00:57:26.8092899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8093011Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8093290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8093370Z return func(*args, **kwargs) 2025-10-10T00:57:26.8093664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8093757Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8093761Z 2025-10-10T00:57:26.8093885Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8094106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8094179Z res = mod(**inputs) 2025-10-10T00:57:26.8094476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8094552Z outputs = self.model( 2025-10-10T00:57:26.8094849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8094930Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8095220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8095309Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8095570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8095665Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8095929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8096022Z return func(*args, **kwargs) 2025-10-10T00:57:26.8096309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8096418Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8096691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8096803Z return func(*args, **kwargs) 2025-10-10T00:57:26.8097090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8097187Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8097191Z 2025-10-10T00:57:26.8097281Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8097404Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8097623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8097702Z res = mod(**inputs) 2025-10-10T00:57:26.8097985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8098060Z outputs = self.model( 2025-10-10T00:57:26.8098388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8098470Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8098759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8098841Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8099092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8099181Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8099451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8099536Z return func(*args, **kwargs) 2025-10-10T00:57:26.8099812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8099927Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8100196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8100274Z return func(*args, **kwargs) 2025-10-10T00:57:26.8100559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8100669Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8100996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8101148Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8101151Z 2025-10-10T00:57:26.8101274Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8101491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8101563Z res = mod(**inputs) 2025-10-10T00:57:26.8101857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8101933Z outputs = self.model( 2025-10-10T00:57:26.8102239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8102321Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8102602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8102706Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8102953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8103049Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8103312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8103426Z return func(*args, **kwargs) 2025-10-10T00:57:26.8103711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8103821Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8104095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8104170Z return func(*args, **kwargs) 2025-10-10T00:57:26.8104455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8104547Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8104551Z 2025-10-10T00:57:26.8104664Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8104891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8104963Z res = mod(**inputs) 2025-10-10T00:57:26.8105252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8105328Z outputs = self.model( 2025-10-10T00:57:26.8105609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8105697Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8105978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8106064Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8106392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8106491Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8106769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8106849Z return func(*args, **kwargs) 2025-10-10T00:57:26.8107138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-10-10T00:57:26.8107231Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.8107235Z 2025-10-10T00:57:26.8107358Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8107578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8107653Z res = mod(**inputs) 2025-10-10T00:57:26.8107944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8108020Z outputs = self.model( 2025-10-10T00:57:26.8108310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8108395Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8108675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8108790Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8109036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8109133Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8109430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8109519Z return func(*args, **kwargs) 2025-10-10T00:57:26.8109798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8109918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8110216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8110311Z return func(*args, **kwargs) 2025-10-10T00:57:26.8110602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8110772Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8110776Z 2025-10-10T00:57:26.8110893Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8111123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8111196Z res = mod(**inputs) 2025-10-10T00:57:26.8111501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8111580Z outputs = self.model( 2025-10-10T00:57:26.8111889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8111972Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8112252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8112344Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8112593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8112689Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8112961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8113039Z return func(*args, **kwargs) 2025-10-10T00:57:26.8113324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8113444Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8113739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8113816Z return func(*args, **kwargs) 2025-10-10T00:57:26.8114091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8114189Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8114192Z 2025-10-10T00:57:26.8114303Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8114528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8114598Z res = mod(**inputs) 2025-10-10T00:57:26.8114884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8114958Z outputs = self.model( 2025-10-10T00:57:26.8115233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8115321Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8115594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8115707Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8115942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8116028Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8116314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8116393Z return func(*args, **kwargs) 2025-10-10T00:57:26.8116673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8116828Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8117108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8117183Z return func(*args, **kwargs) 2025-10-10T00:57:26.8117455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8117557Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8117561Z 2025-10-10T00:57:26.8117651Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8117770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8117984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8118052Z res = mod(**inputs) 2025-10-10T00:57:26.8118338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8118415Z outputs = self.model( 2025-10-10T00:57:26.8118698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8118778Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8119052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8119135Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8119376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8119467Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8119730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8119811Z return func(*args, **kwargs) 2025-10-10T00:57:26.8120084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8120201Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8120467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8120543Z return func(*args, **kwargs) 2025-10-10T00:57:26.8120820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8120928Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8121240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8121392Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8121396Z 2025-10-10T00:57:26.8121506Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8121729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8121798Z res = mod(**inputs) 2025-10-10T00:57:26.8122080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8122174Z outputs = self.model( 2025-10-10T00:57:26.8122441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8122527Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8122826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8122912Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8123148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8123251Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8123545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8123619Z return func(*args, **kwargs) 2025-10-10T00:57:26.8123910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8124024Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8124293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8124374Z return func(*args, **kwargs) 2025-10-10T00:57:26.8124640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8124738Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8124741Z 2025-10-10T00:57:26.8124854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8125077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8125146Z res = mod(**inputs) 2025-10-10T00:57:26.8125436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8125519Z outputs = self.model( 2025-10-10T00:57:26.8125799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8125886Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8126212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8126292Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8126543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8126637Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8126927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8127005Z return func(*args, **kwargs) 2025-10-10T00:57:26.8127288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8127420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8127424Z 2025-10-10T00:57:26.8127540Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8127769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8127841Z res = mod(**inputs) 2025-10-10T00:57:26.8128149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8128226Z outputs = self.model( 2025-10-10T00:57:26.8128512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8128617Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8128887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8128971Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8129224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8129311Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8129587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8129660Z return func(*args, **kwargs) 2025-10-10T00:57:26.8129952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8130095Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8130099Z 2025-10-10T00:57:26.8130216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8130429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8130498Z res = mod(**inputs) 2025-10-10T00:57:26.8130778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8130854Z outputs = self.model( 2025-10-10T00:57:26.8131131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8131208Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8131610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8131711Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8131951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8132047Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8132305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8132390Z return func(*args, **kwargs) 2025-10-10T00:57:26.8132660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8132749Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8132753Z 2025-10-10T00:57:26.8132873Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8133084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8133164Z res = mod(**inputs) 2025-10-10T00:57:26.8133437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8133512Z outputs = self.model( 2025-10-10T00:57:26.8133791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8133868Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8134147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8134227Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8134464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8134559Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8134817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8134901Z return func(*args, **kwargs) 2025-10-10T00:57:26.8135177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8135367Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8135609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8135679Z return func(*args, **kwargs) 2025-10-10T00:57:26.8135967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8136121Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8136125Z 2025-10-10T00:57:26.8136238Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8136464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8136570Z res = mod(**inputs) 2025-10-10T00:57:26.8136834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8136906Z outputs = self.model( 2025-10-10T00:57:26.8137169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8137243Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8137504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8137579Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8137802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8137889Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8138136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8138213Z return func(*args, **kwargs) 2025-10-10T00:57:26.8138466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8138570Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8138821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8138894Z return func(*args, **kwargs) 2025-10-10T00:57:26.8139156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8139240Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8139243Z 2025-10-10T00:57:26.8139353Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8139555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8139623Z res = mod(**inputs) 2025-10-10T00:57:26.8139886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8139958Z outputs = self.model( 2025-10-10T00:57:26.8140222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8140295Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8140551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8140635Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8140859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8140947Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8141196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8141266Z return func(*args, **kwargs) 2025-10-10T00:57:26.8141565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8141671Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8141936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8142030Z return func(*args, **kwargs) 2025-10-10T00:57:26.8142309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8142404Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8142407Z 2025-10-10T00:57:26.8142498Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8142652Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8142864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8142943Z res = mod(**inputs) 2025-10-10T00:57:26.8143215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8143289Z outputs = self.model( 2025-10-10T00:57:26.8143568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8143647Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8143929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8144008Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8144253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8144350Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8144615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8144700Z return func(*args, **kwargs) 2025-10-10T00:57:26.8144976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8145093Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8145358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8145436Z return func(*args, **kwargs) 2025-10-10T00:57:26.8145721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8145830Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8146161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8146376Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8146386Z 2025-10-10T00:57:26.8146506Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8146732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8146802Z res = mod(**inputs) 2025-10-10T00:57:26.8147089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8147165Z outputs = self.model( 2025-10-10T00:57:26.8147457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8147538Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8147819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8147909Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8148182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8148275Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8148520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8148608Z return func(*args, **kwargs) 2025-10-10T00:57:26.8148873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8148974Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8149228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8149333Z return func(*args, **kwargs) 2025-10-10T00:57:26.8149590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8149685Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8149688Z 2025-10-10T00:57:26.8149795Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8150003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8150068Z res = mod(**inputs) 2025-10-10T00:57:26.8150337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8150407Z outputs = self.model( 2025-10-10T00:57:26.8150662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8150746Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8151004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8151087Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8151313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8151394Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8151648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8151719Z return func(*args, **kwargs) 2025-10-10T00:57:26.8151982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8152093Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8152347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8152418Z return func(*args, **kwargs) 2025-10-10T00:57:26.8152671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8152833Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8152837Z 2025-10-10T00:57:26.8152941Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8153150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8153216Z res = mod(**inputs) 2025-10-10T00:57:26.8153474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8153550Z outputs = self.model( 2025-10-10T00:57:26.8153809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8153890Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8154146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8154247Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8154471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8154562Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8154823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8154894Z return func(*args, **kwargs) 2025-10-10T00:57:26.8155145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8155252Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8155519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8155600Z return func(*args, **kwargs) 2025-10-10T00:57:26.8155852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8155942Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8155945Z 2025-10-10T00:57:26.8156048Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8156245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8156318Z res = mod(**inputs) 2025-10-10T00:57:26.8156580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8156657Z outputs = self.model( 2025-10-10T00:57:26.8156917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8157004Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8157276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8157357Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8157602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8157689Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8157962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8158038Z return func(*args, **kwargs) 2025-10-10T00:57:26.8158310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8158429Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8158676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8158756Z return func(*args, **kwargs) 2025-10-10T00:57:26.8159010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8159100Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8159111Z 2025-10-10T00:57:26.8159197Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8159305Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8159516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8159583Z res = mod(**inputs) 2025-10-10T00:57:26.8159847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8159921Z outputs = self.model( 2025-10-10T00:57:26.8160181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8160285Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8160554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8160641Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8160900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8160983Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8161236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8161307Z return func(*args, **kwargs) 2025-10-10T00:57:26.8161595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8161720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8161972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8162044Z return func(*args, **kwargs) 2025-10-10T00:57:26.8162298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8162406Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8162700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8162841Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8162844Z 2025-10-10T00:57:26.8162947Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8163152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8163230Z res = mod(**inputs) 2025-10-10T00:57:26.8163501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8163584Z outputs = self.model( 2025-10-10T00:57:26.8163896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8163979Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8164237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8164312Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8164545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8164626Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8164879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8164951Z return func(*args, **kwargs) 2025-10-10T00:57:26.8165210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8165335Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8165604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8165686Z return func(*args, **kwargs) 2025-10-10T00:57:26.8165951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8166039Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8166051Z 2025-10-10T00:57:26.8166162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8166376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8166453Z res = mod(**inputs) 2025-10-10T00:57:26.8166754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8166834Z outputs = self.model( 2025-10-10T00:57:26.8167138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8167235Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8167503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8167576Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8167819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8167939Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8168209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8168293Z return func(*args, **kwargs) 2025-10-10T00:57:26.8168559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-10-10T00:57:26.8168653Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.8168657Z 2025-10-10T00:57:26.8168767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8168986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8169054Z res = mod(**inputs) 2025-10-10T00:57:26.8169328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8169410Z outputs = self.model( 2025-10-10T00:57:26.8169698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8169783Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8170071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8170149Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8170392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8170482Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8170752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8170827Z return func(*args, **kwargs) 2025-10-10T00:57:26.8171094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8171235Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8171239Z 2025-10-10T00:57:26.8171349Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8171567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8171636Z res = mod(**inputs) 2025-10-10T00:57:26.8171925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8172000Z outputs = self.model( 2025-10-10T00:57:26.8172281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8172368Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8172640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8172729Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8172966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8173073Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8173339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8173414Z return func(*args, **kwargs) 2025-10-10T00:57:26.8173706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8173836Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8173841Z 2025-10-10T00:57:26.8173959Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8174175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8174243Z res = mod(**inputs) 2025-10-10T00:57:26.8174558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8174633Z outputs = self.model( 2025-10-10T00:57:26.8174923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8175001Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8175272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8175361Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8175601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8175695Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8175967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8176046Z return func(*args, **kwargs) 2025-10-10T00:57:26.8176324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8176421Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8176425Z 2025-10-10T00:57:26.8176538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8176739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8176813Z res = mod(**inputs) 2025-10-10T00:57:26.8177073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8177143Z outputs = self.model( 2025-10-10T00:57:26.8177412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8177485Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8177754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8177829Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8178054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8178143Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8178391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8178468Z return func(*args, **kwargs) 2025-10-10T00:57:26.8178724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8178827Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8179081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8179154Z return func(*args, **kwargs) 2025-10-10T00:57:26.8179415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8179599Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8179603Z 2025-10-10T00:57:26.8179726Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8180965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8181049Z res = mod(**inputs) 2025-10-10T00:57:26.8181337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8181411Z outputs = self.model( 2025-10-10T00:57:26.8181711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8181806Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8182073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8182161Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8182395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8182489Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8182749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8182833Z return func(*args, **kwargs) 2025-10-10T00:57:26.8183100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8183208Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8183478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8183551Z return func(*args, **kwargs) 2025-10-10T00:57:26.8183825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8183914Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8183917Z 2025-10-10T00:57:26.8184029Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8184249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8184316Z res = mod(**inputs) 2025-10-10T00:57:26.8184595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8184669Z outputs = self.model( 2025-10-10T00:57:26.8184959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8185040Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8185319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8185407Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8185648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8185743Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8186008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8186085Z return func(*args, **kwargs) 2025-10-10T00:57:26.8186451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8186567Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8186847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8186926Z return func(*args, **kwargs) 2025-10-10T00:57:26.8187224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8187329Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8187333Z 2025-10-10T00:57:26.8187426Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8187574Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8187803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8187878Z res = mod(**inputs) 2025-10-10T00:57:26.8188140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8188245Z outputs = self.model( 2025-10-10T00:57:26.8188513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8188589Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8188853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8188926Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8189154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8189244Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8189488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8189567Z return func(*args, **kwargs) 2025-10-10T00:57:26.8189830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8189937Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8190202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8190278Z return func(*args, **kwargs) 2025-10-10T00:57:26.8190563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8190671Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8191000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8191148Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8191152Z 2025-10-10T00:57:26.8191267Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8191495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8191568Z res = mod(**inputs) 2025-10-10T00:57:26.8191860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8191937Z outputs = self.model( 2025-10-10T00:57:26.8192216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8192304Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8192587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8192675Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8192918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8193012Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8193282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8193359Z return func(*args, **kwargs) 2025-10-10T00:57:26.8193665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-10-10T00:57:26.8193774Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:26.8194045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8194142Z return func(*args, **kwargs) 2025-10-10T00:57:26.8194424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8194524Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8194528Z 2025-10-10T00:57:26.8194642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8194901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8194975Z res = mod(**inputs) 2025-10-10T00:57:26.8195260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8195337Z outputs = self.model( 2025-10-10T00:57:26.8195616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8195705Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8195988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8196075Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8196321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8196412Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8196686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8196765Z return func(*args, **kwargs) 2025-10-10T00:57:26.8197048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8197171Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8197437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8197521Z return func(*args, **kwargs) 2025-10-10T00:57:26.8197795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-10-10T00:57:26.8197968Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:26.8197974Z 2025-10-10T00:57:26.8198090Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8198317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8198392Z res = mod(**inputs) 2025-10-10T00:57:26.8198676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8198758Z outputs = self.model( 2025-10-10T00:57:26.8199038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8199125Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8199404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8199483Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8199735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8199824Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8200097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8200193Z return func(*args, **kwargs) 2025-10-10T00:57:26.8200477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8200597Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8200885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8200969Z return func(*args, **kwargs) 2025-10-10T00:57:26.8201249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-10-10T00:57:26.8201349Z key_states = self.k_proj(current_states) 2025-10-10T00:57:26.8201385Z 2025-10-10T00:57:26.8201503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8201723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8201807Z res = mod(**inputs) 2025-10-10T00:57:26.8202099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8202186Z outputs = self.model( 2025-10-10T00:57:26.8202479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8202562Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8202859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8202943Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8203205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8203300Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8203585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8203668Z return func(*args, **kwargs) 2025-10-10T00:57:26.8203957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8204092Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8204371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8204458Z return func(*args, **kwargs) 2025-10-10T00:57:26.8204747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-10-10T00:57:26.8204852Z value_states = self.v_proj(current_states) 2025-10-10T00:57:26.8204863Z 2025-10-10T00:57:26.8204958Z cudagraph partition due to non gpu ops 2025-10-10T00:57:26.8205076Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8205316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8205392Z res = mod(**inputs) 2025-10-10T00:57:26.8205695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8205777Z outputs = self.model( 2025-10-10T00:57:26.8206068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8206161Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8206452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8206547Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8206800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8206910Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8207183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8207260Z return func(*args, **kwargs) 2025-10-10T00:57:26.8207560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8207680Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8207945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8208027Z return func(*args, **kwargs) 2025-10-10T00:57:26.8208320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-10-10T00:57:26.8208460Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:26.8208779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:26.8208939Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:26.8208943Z 2025-10-10T00:57:26.8209055Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8209277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8209358Z res = mod(**inputs) 2025-10-10T00:57:26.8209647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8209728Z outputs = self.model( 2025-10-10T00:57:26.8209999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8210078Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8210356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8210436Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8210678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8210766Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8211037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8211115Z return func(*args, **kwargs) 2025-10-10T00:57:26.8211390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-10-10T00:57:26.8211518Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-10-10T00:57:26.8211785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8211869Z return func(*args, **kwargs) 2025-10-10T00:57:26.8212143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-10-10T00:57:26.8212235Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:26.8212239Z 2025-10-10T00:57:26.8212360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8212580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8212659Z res = mod(**inputs) 2025-10-10T00:57:26.8212939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8213014Z outputs = self.model( 2025-10-10T00:57:26.8213304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8213384Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8213699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8213779Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8214030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8214137Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8214406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8214490Z return func(*args, **kwargs) 2025-10-10T00:57:26.8214791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8214966Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8214971Z 2025-10-10T00:57:26.8215087Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8215307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8215386Z res = mod(**inputs) 2025-10-10T00:57:26.8215667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8215747Z outputs = self.model( 2025-10-10T00:57:26.8216049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8216134Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8216435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8216516Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8216770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8216858Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8217133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8217211Z return func(*args, **kwargs) 2025-10-10T00:57:26.8217513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-10-10T00:57:26.8217654Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:26.8217658Z 2025-10-10T00:57:26.8217771Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8217997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8218067Z res = mod(**inputs) 2025-10-10T00:57:26.8218354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8218436Z outputs = self.model( 2025-10-10T00:57:26.8218718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8218807Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8219107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8219196Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8219443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8219532Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8219810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8219889Z return func(*args, **kwargs) 2025-10-10T00:57:26.8220173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-10-10T00:57:26.8220284Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:26.8220287Z 2025-10-10T00:57:26.8220401Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8220625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8220696Z res = mod(**inputs) 2025-10-10T00:57:26.8221003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-10-10T00:57:26.8221081Z outputs = self.model( 2025-10-10T00:57:26.8221384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-10-10T00:57:26.8221465Z decoder_outputs = self.decoder( 2025-10-10T00:57:26.8221778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-10-10T00:57:26.8221871Z layer_outputs = decoder_layer( 2025-10-10T00:57:26.8222115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:26.8222212Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:26.8222479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:26.8222557Z return func(*args, **kwargs) 2025-10-10T00:57:26.8222842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-10-10T00:57:26.8222934Z hidden_states = residual + hidden_states 2025-10-10T00:57:26.8222937Z 2025-10-10T00:57:26.8223057Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8223278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8223349Z res = mod(**inputs) 2025-10-10T00:57:26.8223637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1420, in forward 2025-10-10T00:57:26.8223729Z lm_logits = self.lm_head(outputs[0]) 2025-10-10T00:57:26.8223733Z 2025-10-10T00:57:26.8223855Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:26.8224072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:26.8224152Z res = mod(**inputs) 2025-10-10T00:57:26.8224434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1427, in forward 2025-10-10T00:57:26.8224624Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:57:26.8224629Z 2025-10-10T00:57:38.9663686Z Compilation time (from dynamo_timed): 27.797818347 2025-10-10T00:57:38.9783663Z pass 2025-10-10T00:57:38.9784125Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:38.9785040Z TIMING: _recursive_pre_grad_passes:0.01458 _recursive_joint_graph_passes:0.84186 _recursive_post_grad_passes:0.15348 async_compile.wait:0.71166 code_gen:11.64148 inductor_compile:14.83328 backend_compile:21.88453 gc:0.00047 entire_frame_compile:27.79782 total_wall_time:27.79782 2025-10-10T00:57:38.9786076Z STATS: call_* op count: 906 | FakeTensorMode.__torch_dispatch__:18378 | FakeTensor.__torch_dispatch__:10222 | ProxyTorchDispatchMode.__torch_dispatch__:4932 2025-10-10T00:57:38.9786851Z Dynamo produced 1 graphs covering 906 ops with 0 graph breaks (0 unique) 2025-10-10T00:57:42.3596736Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:57:42.3597625Z import pynvml # type: ignore[import] 2025-10-10T00:57:45.8151220Z 2025-10-10T00:57:48.5370680Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:57:48.5372491Z loading model: 0it [00:02, ?it/s] 2025-10-10T00:57:48.5381948Z cpu eval MBartForCausalLM 2025-10-10T00:57:50.2533088Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:50.9419118Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:51.6469145Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:57:59.4660021Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4660379Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4660613Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4661292Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4661531Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4661762Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4661990Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4662220Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4662449Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4663248Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4663997Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4664319Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4664647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4665097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4665489Z res = mod(**inputs) 2025-10-10T00:57:59.4665948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4666736Z outputs = self.model.decoder( 2025-10-10T00:57:59.4667204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4667726Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4668137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4668553Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4669006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4669432Z return func(*args, **kwargs) 2025-10-10T00:57:59.4669861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4670334Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4670918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4671341Z return func(*args, **kwargs) 2025-10-10T00:57:59.4671777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4672304Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4672536Z 2025-10-10T00:57:59.4672670Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4673092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4673450Z res = mod(**inputs) 2025-10-10T00:57:59.4673859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4674298Z outputs = self.model.decoder( 2025-10-10T00:57:59.4674732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4675177Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4675569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4676229Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4676674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4677122Z return func(*args, **kwargs) 2025-10-10T00:57:59.4677599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4678052Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4678502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4678919Z return func(*args, **kwargs) 2025-10-10T00:57:59.4679411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4679874Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4680041Z 2025-10-10T00:57:59.4680163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4680574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4680952Z res = mod(**inputs) 2025-10-10T00:57:59.4681351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4681772Z outputs = self.model.decoder( 2025-10-10T00:57:59.4682185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4682616Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4683008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4683398Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4683817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4684241Z return func(*args, **kwargs) 2025-10-10T00:57:59.4684657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4685125Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4685579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4686028Z return func(*args, **kwargs) 2025-10-10T00:57:59.4686431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4686880Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4687039Z 2025-10-10T00:57:59.4687136Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4687395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4687790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4688141Z res = mod(**inputs) 2025-10-10T00:57:59.4688544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4688969Z outputs = self.model.decoder( 2025-10-10T00:57:59.4689387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4689885Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4690269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4690674Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4691097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4691513Z return func(*args, **kwargs) 2025-10-10T00:57:59.4691971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4692438Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4692901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4693315Z return func(*args, **kwargs) 2025-10-10T00:57:59.4693721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4694188Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4694716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4695290Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4695509Z 2025-10-10T00:57:59.4695634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4696046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4696411Z res = mod(**inputs) 2025-10-10T00:57:59.4696831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4697263Z outputs = self.model.decoder( 2025-10-10T00:57:59.4697697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4698133Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4698531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4698947Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4699373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4699796Z return func(*args, **kwargs) 2025-10-10T00:57:59.4700215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4700677Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4701123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4701537Z return func(*args, **kwargs) 2025-10-10T00:57:59.4701953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4702401Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4702559Z 2025-10-10T00:57:59.4702686Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4703085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4703448Z res = mod(**inputs) 2025-10-10T00:57:59.4703856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4704382Z outputs = self.model.decoder( 2025-10-10T00:57:59.4704821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4705275Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4705677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4706102Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4706657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4707089Z return func(*args, **kwargs) 2025-10-10T00:57:59.4707518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4708079Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4708277Z 2025-10-10T00:57:59.4708407Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4708807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4709190Z res = mod(**inputs) 2025-10-10T00:57:59.4709598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4710048Z outputs = self.model.decoder( 2025-10-10T00:57:59.4710492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4710988Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4711367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4711794Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4712265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4712688Z return func(*args, **kwargs) 2025-10-10T00:57:59.4713102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4713593Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4714031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4714420Z return self.act(input) 2025-10-10T00:57:59.4714548Z 2025-10-10T00:57:59.4714681Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4715073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4715435Z res = mod(**inputs) 2025-10-10T00:57:59.4715843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4716284Z outputs = self.model.decoder( 2025-10-10T00:57:59.4716714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4717145Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4717538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4717950Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4718372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4718790Z return func(*args, **kwargs) 2025-10-10T00:57:59.4719222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4719679Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4719837Z 2025-10-10T00:57:59.4719966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4720378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4720743Z res = mod(**inputs) 2025-10-10T00:57:59.4721176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4721632Z outputs = self.model.decoder( 2025-10-10T00:57:59.4722056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4722495Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4722889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4723311Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4723822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4724249Z return func(*args, **kwargs) 2025-10-10T00:57:59.4724676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4725155Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4725596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4726018Z return func(*args, **kwargs) 2025-10-10T00:57:59.4726452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4726975Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4727208Z 2025-10-10T00:57:59.4727323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4727712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4728062Z res = mod(**inputs) 2025-10-10T00:57:59.4728454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4728869Z outputs = self.model.decoder( 2025-10-10T00:57:59.4729282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4729746Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4730131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4730525Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4730940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4731341Z return func(*args, **kwargs) 2025-10-10T00:57:59.4731953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4732413Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4732852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4733272Z return func(*args, **kwargs) 2025-10-10T00:57:59.4733687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4734126Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4734284Z 2025-10-10T00:57:59.4734412Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4734799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4735158Z res = mod(**inputs) 2025-10-10T00:57:59.4735561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4735987Z outputs = self.model.decoder( 2025-10-10T00:57:59.4736398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4736826Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4737210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4737610Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4738024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4738429Z return func(*args, **kwargs) 2025-10-10T00:57:59.4738839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4739381Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4739811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4740222Z return func(*args, **kwargs) 2025-10-10T00:57:59.4740622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4741043Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4741207Z 2025-10-10T00:57:59.4741300Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4741569Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4741985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4742368Z res = mod(**inputs) 2025-10-10T00:57:59.4742763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4743189Z outputs = self.model.decoder( 2025-10-10T00:57:59.4743600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4744011Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4744398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4744818Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4745246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4745665Z return func(*args, **kwargs) 2025-10-10T00:57:59.4746098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4746618Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4747079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4747500Z return func(*args, **kwargs) 2025-10-10T00:57:59.4747899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4748351Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4748830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4749335Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4749527Z 2025-10-10T00:57:59.4749647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4750016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4750369Z res = mod(**inputs) 2025-10-10T00:57:59.4750761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4751188Z outputs = self.model.decoder( 2025-10-10T00:57:59.4751592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4752021Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4752383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4752763Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4753153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4753528Z return func(*args, **kwargs) 2025-10-10T00:57:59.4753915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4754376Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4754785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4755163Z return func(*args, **kwargs) 2025-10-10T00:57:59.4755556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4755968Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4756117Z 2025-10-10T00:57:59.4756229Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4756604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4756947Z res = mod(**inputs) 2025-10-10T00:57:59.4757322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4757720Z outputs = self.model.decoder( 2025-10-10T00:57:59.4758114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4758515Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4758868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4759243Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4759634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4760007Z return func(*args, **kwargs) 2025-10-10T00:57:59.4760379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4760809Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4760992Z 2025-10-10T00:57:59.4761098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4761467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4761793Z res = mod(**inputs) 2025-10-10T00:57:59.4762156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4762557Z outputs = self.model.decoder( 2025-10-10T00:57:59.4762949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4763348Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4763721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4764099Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4764505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4764914Z return func(*args, **kwargs) 2025-10-10T00:57:59.4765321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4765796Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4766230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4766616Z return self.act(input) 2025-10-10T00:57:59.4766747Z 2025-10-10T00:57:59.4766865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4767265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4767619Z res = mod(**inputs) 2025-10-10T00:57:59.4768032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4768455Z outputs = self.model.decoder( 2025-10-10T00:57:59.4768900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4769320Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4769694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4770160Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4770587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4771024Z return func(*args, **kwargs) 2025-10-10T00:57:59.4771430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4771912Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4772070Z 2025-10-10T00:57:59.4772184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4772584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4772944Z res = mod(**inputs) 2025-10-10T00:57:59.4773338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4773785Z outputs = self.model.decoder( 2025-10-10T00:57:59.4774242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4774684Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4775075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4775480Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4775917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4776339Z return func(*args, **kwargs) 2025-10-10T00:57:59.4776759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.4777190Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.4777351Z 2025-10-10T00:57:59.4777468Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4777870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4778225Z res = mod(**inputs) 2025-10-10T00:57:59.4778623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4779055Z outputs = self.model.decoder( 2025-10-10T00:57:59.4779490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4779926Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4780316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4780732Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4781156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4781570Z return func(*args, **kwargs) 2025-10-10T00:57:59.4781988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4782452Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4782901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4783321Z return func(*args, **kwargs) 2025-10-10T00:57:59.4783736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4784254Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4784509Z 2025-10-10T00:57:59.4784634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4785026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4785404Z res = mod(**inputs) 2025-10-10T00:57:59.4785831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4786281Z outputs = self.model.decoder( 2025-10-10T00:57:59.4786819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4787250Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4787705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4788115Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4788542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4788969Z return func(*args, **kwargs) 2025-10-10T00:57:59.4789385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4789842Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4790286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4790699Z return func(*args, **kwargs) 2025-10-10T00:57:59.4791107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4791550Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4791710Z 2025-10-10T00:57:59.4791836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4792225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4792571Z res = mod(**inputs) 2025-10-10T00:57:59.4792957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4793376Z outputs = self.model.decoder( 2025-10-10T00:57:59.4793788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4794203Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4794572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4794969Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4795400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4795811Z return func(*args, **kwargs) 2025-10-10T00:57:59.4796228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4796675Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4797118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4797538Z return func(*args, **kwargs) 2025-10-10T00:57:59.4797937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4798368Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4798522Z 2025-10-10T00:57:59.4798612Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4798878Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4799266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4799649Z res = mod(**inputs) 2025-10-10T00:57:59.4800037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4800464Z outputs = self.model.decoder( 2025-10-10T00:57:59.4800909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4801347Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4801744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4802154Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4802591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4803015Z return func(*args, **kwargs) 2025-10-10T00:57:59.4803422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4803865Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4804302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4804703Z return func(*args, **kwargs) 2025-10-10T00:57:59.4805112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4805567Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4806059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4806611Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4806816Z 2025-10-10T00:57:59.4806930Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4807321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4807666Z res = mod(**inputs) 2025-10-10T00:57:59.4808052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4808479Z outputs = self.model.decoder( 2025-10-10T00:57:59.4808897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4809320Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4809694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4810094Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4810513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4810915Z return func(*args, **kwargs) 2025-10-10T00:57:59.4811326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4811766Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4812208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4812629Z return func(*args, **kwargs) 2025-10-10T00:57:59.4813053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4813500Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4813654Z 2025-10-10T00:57:59.4813769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4814174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4814541Z res = mod(**inputs) 2025-10-10T00:57:59.4814935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4815385Z outputs = self.model.decoder( 2025-10-10T00:57:59.4815810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4816245Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4816662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4817078Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4817501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4817924Z return func(*args, **kwargs) 2025-10-10T00:57:59.4818389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4818875Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4819070Z 2025-10-10T00:57:59.4819198Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4819601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4819974Z res = mod(**inputs) 2025-10-10T00:57:59.4820381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4820813Z outputs = self.model.decoder( 2025-10-10T00:57:59.4821240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4821682Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4822107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4822524Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4822954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4823374Z return func(*args, **kwargs) 2025-10-10T00:57:59.4823792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4824275Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4824711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4825101Z return self.act(input) 2025-10-10T00:57:59.4825225Z 2025-10-10T00:57:59.4825342Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4825750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4826109Z res = mod(**inputs) 2025-10-10T00:57:59.4826610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4827047Z outputs = self.model.decoder( 2025-10-10T00:57:59.4827487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4827928Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4828322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4828737Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4829156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4829569Z return func(*args, **kwargs) 2025-10-10T00:57:59.4829970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4830398Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4830548Z 2025-10-10T00:57:59.4830704Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4831080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4831430Z res = mod(**inputs) 2025-10-10T00:57:59.4832084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4832519Z outputs = self.model.decoder( 2025-10-10T00:57:59.4832926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4833342Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4833766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4834187Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4834601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4834997Z return func(*args, **kwargs) 2025-10-10T00:57:59.4835403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4835849Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4836283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4836680Z return func(*args, **kwargs) 2025-10-10T00:57:59.4837087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4837591Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4837816Z 2025-10-10T00:57:59.4837941Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4838332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4838669Z res = mod(**inputs) 2025-10-10T00:57:59.4839042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4839438Z outputs = self.model.decoder( 2025-10-10T00:57:59.4839832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4840226Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4840578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4840950Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4841340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4841721Z return func(*args, **kwargs) 2025-10-10T00:57:59.4842093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4842517Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4842922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4843302Z return func(*args, **kwargs) 2025-10-10T00:57:59.4843680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4844076Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4844223Z 2025-10-10T00:57:59.4844332Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4844701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4845048Z res = mod(**inputs) 2025-10-10T00:57:59.4845437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4845881Z outputs = self.model.decoder( 2025-10-10T00:57:59.4846292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4846719Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4847093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4847464Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4847876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4848278Z return func(*args, **kwargs) 2025-10-10T00:57:59.4848725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4849162Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4849561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4849939Z return func(*args, **kwargs) 2025-10-10T00:57:59.4850319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4850728Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4850872Z 2025-10-10T00:57:59.4850962Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4851205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4851571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4851901Z res = mod(**inputs) 2025-10-10T00:57:59.4852270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4852658Z outputs = self.model.decoder( 2025-10-10T00:57:59.4853050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4853456Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4853835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4854231Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4854638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4855037Z return func(*args, **kwargs) 2025-10-10T00:57:59.4855443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4855894Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4856314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4856718Z return func(*args, **kwargs) 2025-10-10T00:57:59.4857124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4857571Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4858056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4858570Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4858777Z 2025-10-10T00:57:59.4858890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4859283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4859634Z res = mod(**inputs) 2025-10-10T00:57:59.4860024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4860481Z outputs = self.model.decoder( 2025-10-10T00:57:59.4860894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4861314Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4861725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4862139Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4862559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4862972Z return func(*args, **kwargs) 2025-10-10T00:57:59.4863419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4863884Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4864330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4864751Z return func(*args, **kwargs) 2025-10-10T00:57:59.4865168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4865609Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4865765Z 2025-10-10T00:57:59.4865890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4866285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4866728Z res = mod(**inputs) 2025-10-10T00:57:59.4867144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4867582Z outputs = self.model.decoder( 2025-10-10T00:57:59.4868007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4868431Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4868814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4869214Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4869634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4870013Z return func(*args, **kwargs) 2025-10-10T00:57:59.4870394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4870825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4871003Z 2025-10-10T00:57:59.4871120Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4871487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4871815Z res = mod(**inputs) 2025-10-10T00:57:59.4872187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4872585Z outputs = self.model.decoder( 2025-10-10T00:57:59.4872975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4873369Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4873731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4874108Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4874507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4874886Z return func(*args, **kwargs) 2025-10-10T00:57:59.4875259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4875721Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4876135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4876513Z return self.act(input) 2025-10-10T00:57:59.4876634Z 2025-10-10T00:57:59.4876773Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4877154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4877502Z res = mod(**inputs) 2025-10-10T00:57:59.4877892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4878391Z outputs = self.model.decoder( 2025-10-10T00:57:59.4878801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4879202Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4879561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4879950Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4880365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4880760Z return func(*args, **kwargs) 2025-10-10T00:57:59.4881164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4881588Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4881745Z 2025-10-10T00:57:59.4881868Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4882239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4882563Z res = mod(**inputs) 2025-10-10T00:57:59.4882934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4883329Z outputs = self.model.decoder( 2025-10-10T00:57:59.4883724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4884138Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4884517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4884936Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4885324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4885710Z return func(*args, **kwargs) 2025-10-10T00:57:59.4886086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.4886491Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.4886636Z 2025-10-10T00:57:59.4886744Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4887107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4887428Z res = mod(**inputs) 2025-10-10T00:57:59.4887811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4888204Z outputs = self.model.decoder( 2025-10-10T00:57:59.4888590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4888983Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4889336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4889707Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4890123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4890526Z return func(*args, **kwargs) 2025-10-10T00:57:59.4890933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4891403Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4891842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4892249Z return func(*args, **kwargs) 2025-10-10T00:57:59.4892675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4893186Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4893410Z 2025-10-10T00:57:59.4893526Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4893921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4894267Z res = mod(**inputs) 2025-10-10T00:57:59.4894656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4895069Z outputs = self.model.decoder( 2025-10-10T00:57:59.4895483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4895904Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4896281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4896681Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4897087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4897496Z return func(*args, **kwargs) 2025-10-10T00:57:59.4897905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4898354Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4898778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4899179Z return func(*args, **kwargs) 2025-10-10T00:57:59.4899585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4900024Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4900177Z 2025-10-10T00:57:59.4900551Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4900950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4901317Z res = mod(**inputs) 2025-10-10T00:57:59.4901720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4902155Z outputs = self.model.decoder( 2025-10-10T00:57:59.4902595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4903029Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4903431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4903840Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4904269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4904723Z return func(*args, **kwargs) 2025-10-10T00:57:59.4905136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4905622Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4906079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4906595Z return func(*args, **kwargs) 2025-10-10T00:57:59.4907041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4907504Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4907671Z 2025-10-10T00:57:59.4907764Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4908038Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4908461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4908840Z res = mod(**inputs) 2025-10-10T00:57:59.4909246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4909682Z outputs = self.model.decoder( 2025-10-10T00:57:59.4910107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4910530Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4910925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4911331Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4911757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4912183Z return func(*args, **kwargs) 2025-10-10T00:57:59.4912601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4913059Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4913506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4913921Z return func(*args, **kwargs) 2025-10-10T00:57:59.4914335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4914786Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4915285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4915826Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4916033Z 2025-10-10T00:57:59.4916160Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4916568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4916918Z res = mod(**inputs) 2025-10-10T00:57:59.4917325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4917780Z outputs = self.model.decoder( 2025-10-10T00:57:59.4918194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4918609Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4918993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4919388Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4919801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4920208Z return func(*args, **kwargs) 2025-10-10T00:57:59.4920609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4921080Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4921511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4921913Z return func(*args, **kwargs) 2025-10-10T00:57:59.4922331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4922764Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4922919Z 2025-10-10T00:57:59.4923033Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4923417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4923761Z res = mod(**inputs) 2025-10-10T00:57:59.4924191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4924612Z outputs = self.model.decoder( 2025-10-10T00:57:59.4925023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4925440Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4925815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4926199Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4926608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4927007Z return func(*args, **kwargs) 2025-10-10T00:57:59.4927410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4927873Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4928066Z 2025-10-10T00:57:59.4928181Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4928567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4928914Z res = mod(**inputs) 2025-10-10T00:57:59.4929308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4929725Z outputs = self.model.decoder( 2025-10-10T00:57:59.4930139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4930558Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4930934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4931332Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4931908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4932324Z return func(*args, **kwargs) 2025-10-10T00:57:59.4932733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4933207Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4933630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4934026Z return self.act(input) 2025-10-10T00:57:59.4934157Z 2025-10-10T00:57:59.4934271Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4934667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4935023Z res = mod(**inputs) 2025-10-10T00:57:59.4935411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4935837Z outputs = self.model.decoder( 2025-10-10T00:57:59.4936316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4936735Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4937105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4937529Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4937948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4938364Z return func(*args, **kwargs) 2025-10-10T00:57:59.4938781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4939311Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4939484Z 2025-10-10T00:57:59.4939596Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4939986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4940345Z res = mod(**inputs) 2025-10-10T00:57:59.4940752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4941185Z outputs = self.model.decoder( 2025-10-10T00:57:59.4941620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4942061Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4942453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4942861Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4943301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4943718Z return func(*args, **kwargs) 2025-10-10T00:57:59.4944137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4944593Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4945027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4945449Z return func(*args, **kwargs) 2025-10-10T00:57:59.4945863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.4946455Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.4946693Z 2025-10-10T00:57:59.4946818Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4947217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4947576Z res = mod(**inputs) 2025-10-10T00:57:59.4947986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4948425Z outputs = self.model.decoder( 2025-10-10T00:57:59.4948849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4949300Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4949696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4950116Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4950543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4950970Z return func(*args, **kwargs) 2025-10-10T00:57:59.4951401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4951885Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4952316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4952716Z return func(*args, **kwargs) 2025-10-10T00:57:59.4953130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.4953558Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.4953712Z 2025-10-10T00:57:59.4953825Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4954217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4954564Z res = mod(**inputs) 2025-10-10T00:57:59.4955025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4955447Z outputs = self.model.decoder( 2025-10-10T00:57:59.4955858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4956269Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4956640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4957016Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4957403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4957785Z return func(*args, **kwargs) 2025-10-10T00:57:59.4958192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4958628Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4959056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4959461Z return func(*args, **kwargs) 2025-10-10T00:57:59.4959863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.4960284Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.4960444Z 2025-10-10T00:57:59.4960536Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.4960799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4961185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4961529Z res = mod(**inputs) 2025-10-10T00:57:59.4961913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4962332Z outputs = self.model.decoder( 2025-10-10T00:57:59.4962742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4963156Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4963530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4963921Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4964335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4964731Z return func(*args, **kwargs) 2025-10-10T00:57:59.4965132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4965543Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4965952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4966328Z return func(*args, **kwargs) 2025-10-10T00:57:59.4966706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.4967150Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.4967595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.4968116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.4968326Z 2025-10-10T00:57:59.4968442Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4968837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4969186Z res = mod(**inputs) 2025-10-10T00:57:59.4969592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4970030Z outputs = self.model.decoder( 2025-10-10T00:57:59.4970437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4970854Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4971225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4971618Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4972031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4972431Z return func(*args, **kwargs) 2025-10-10T00:57:59.4972834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.4973273Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.4973703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4974100Z return func(*args, **kwargs) 2025-10-10T00:57:59.4974500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.4974927Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.4975075Z 2025-10-10T00:57:59.4975188Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4975578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4975925Z res = mod(**inputs) 2025-10-10T00:57:59.4976314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4976726Z outputs = self.model.decoder( 2025-10-10T00:57:59.4977138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4977549Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4977929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4978320Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4978713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4979096Z return func(*args, **kwargs) 2025-10-10T00:57:59.4979473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4979926Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4980112Z 2025-10-10T00:57:59.4980232Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4980616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4980961Z res = mod(**inputs) 2025-10-10T00:57:59.4981353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4981793Z outputs = self.model.decoder( 2025-10-10T00:57:59.4982196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4982616Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4983022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4983451Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4983864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4984259Z return func(*args, **kwargs) 2025-10-10T00:57:59.4984736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.4985218Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.4985654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.4986031Z return self.act(input) 2025-10-10T00:57:59.4986165Z 2025-10-10T00:57:59.4986281Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4986768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4987143Z res = mod(**inputs) 2025-10-10T00:57:59.4987546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4987995Z outputs = self.model.decoder( 2025-10-10T00:57:59.4988387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4988784Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4989145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4989528Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4989914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4990295Z return func(*args, **kwargs) 2025-10-10T00:57:59.4990680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.4991084Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.4991226Z 2025-10-10T00:57:59.4991333Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4991711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4992057Z res = mod(**inputs) 2025-10-10T00:57:59.4992447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4992872Z outputs = self.model.decoder( 2025-10-10T00:57:59.4993252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4993666Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4994047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4994438Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4994839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.4995241Z return func(*args, **kwargs) 2025-10-10T00:57:59.4995638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.4996043Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.4996190Z 2025-10-10T00:57:59.4996340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.4996720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.4997069Z res = mod(**inputs) 2025-10-10T00:57:59.4997475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.4997898Z outputs = self.model.decoder( 2025-10-10T00:57:59.4998306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.4998714Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.4999107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.4999531Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.4999940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5000333Z return func(*args, **kwargs) 2025-10-10T00:57:59.5000743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5001190Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5001620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5002018Z return func(*args, **kwargs) 2025-10-10T00:57:59.5002416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5002919Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5003149Z 2025-10-10T00:57:59.5003263Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5003651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5004000Z res = mod(**inputs) 2025-10-10T00:57:59.5004387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5004808Z outputs = self.model.decoder( 2025-10-10T00:57:59.5005221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5005637Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5006009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5006400Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5006812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5007223Z return func(*args, **kwargs) 2025-10-10T00:57:59.5007625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5008064Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5008494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5008893Z return func(*args, **kwargs) 2025-10-10T00:57:59.5009298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5009724Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5009870Z 2025-10-10T00:57:59.5009983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5010374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5010721Z res = mod(**inputs) 2025-10-10T00:57:59.5011114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5011548Z outputs = self.model.decoder( 2025-10-10T00:57:59.5011953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5012369Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5012813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5013209Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5013614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5014021Z return func(*args, **kwargs) 2025-10-10T00:57:59.5014461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5014910Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5015346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5015740Z return func(*args, **kwargs) 2025-10-10T00:57:59.5016146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5016583Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5016738Z 2025-10-10T00:57:59.5016837Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5017090Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5017478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5017833Z res = mod(**inputs) 2025-10-10T00:57:59.5018255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5018677Z outputs = self.model.decoder( 2025-10-10T00:57:59.5019086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5019505Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5019884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5020324Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5020730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5021137Z return func(*args, **kwargs) 2025-10-10T00:57:59.5021545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5021994Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5022426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5022824Z return func(*args, **kwargs) 2025-10-10T00:57:59.5023241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5023702Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5024205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5024754Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5024966Z 2025-10-10T00:57:59.5025084Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5025486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5025860Z res = mod(**inputs) 2025-10-10T00:57:59.5026264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5026822Z outputs = self.model.decoder( 2025-10-10T00:57:59.5027238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5027671Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5028088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5028500Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5028924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5029358Z return func(*args, **kwargs) 2025-10-10T00:57:59.5029799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5030277Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5030731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5031153Z return func(*args, **kwargs) 2025-10-10T00:57:59.5031681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5032134Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5032292Z 2025-10-10T00:57:59.5032418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5032827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5033196Z res = mod(**inputs) 2025-10-10T00:57:59.5033611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5034061Z outputs = self.model.decoder( 2025-10-10T00:57:59.5034496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5034942Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5035347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5035768Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5036199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5036633Z return func(*args, **kwargs) 2025-10-10T00:57:59.5037052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5037544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5037749Z 2025-10-10T00:57:59.5037868Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5038270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5038627Z res = mod(**inputs) 2025-10-10T00:57:59.5039038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5039481Z outputs = self.model.decoder( 2025-10-10T00:57:59.5039916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5040360Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5040739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5041145Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5041568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5041986Z return func(*args, **kwargs) 2025-10-10T00:57:59.5042397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5042907Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5043326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5043698Z return self.act(input) 2025-10-10T00:57:59.5043817Z 2025-10-10T00:57:59.5043966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5044347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5044694Z res = mod(**inputs) 2025-10-10T00:57:59.5045085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5045563Z outputs = self.model.decoder( 2025-10-10T00:57:59.5045974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5046389Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5046771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5047168Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5047586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5047992Z return func(*args, **kwargs) 2025-10-10T00:57:59.5048393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5048823Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5048980Z 2025-10-10T00:57:59.5049096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5049486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5049826Z res = mod(**inputs) 2025-10-10T00:57:59.5050221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5050640Z outputs = self.model.decoder( 2025-10-10T00:57:59.5051053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5051469Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5051842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5052236Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5052646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5053053Z return func(*args, **kwargs) 2025-10-10T00:57:59.5053452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5053901Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5054337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5054739Z return func(*args, **kwargs) 2025-10-10T00:57:59.5055144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5055642Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5055867Z 2025-10-10T00:57:59.5055980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5056369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5056722Z res = mod(**inputs) 2025-10-10T00:57:59.5057113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5057549Z outputs = self.model.decoder( 2025-10-10T00:57:59.5057963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5058380Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5058775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5059154Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5059559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5059963Z return func(*args, **kwargs) 2025-10-10T00:57:59.5060398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5060865Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5061291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5061697Z return func(*args, **kwargs) 2025-10-10T00:57:59.5062102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5062533Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5062683Z 2025-10-10T00:57:59.5062805Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5063187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5063534Z res = mod(**inputs) 2025-10-10T00:57:59.5063928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5064354Z outputs = self.model.decoder( 2025-10-10T00:57:59.5064772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5065203Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5065595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5066009Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5066519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5066929Z return func(*args, **kwargs) 2025-10-10T00:57:59.5067346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5067803Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5068239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5068620Z return func(*args, **kwargs) 2025-10-10T00:57:59.5068995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5069459Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5069620Z 2025-10-10T00:57:59.5069709Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5069972Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5070356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5070703Z res = mod(**inputs) 2025-10-10T00:57:59.5071092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5071508Z outputs = self.model.decoder( 2025-10-10T00:57:59.5071921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5072329Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5072730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5073124Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5073532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5073931Z return func(*args, **kwargs) 2025-10-10T00:57:59.5074345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5074797Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5075237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5075681Z return func(*args, **kwargs) 2025-10-10T00:57:59.5076083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5076533Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5077022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5077561Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5077565Z 2025-10-10T00:57:59.5077684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5077919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5077992Z res = mod(**inputs) 2025-10-10T00:57:59.5078276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5078371Z outputs = self.model.decoder( 2025-10-10T00:57:59.5078652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5078746Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5079001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5079096Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5079355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5079433Z return func(*args, **kwargs) 2025-10-10T00:57:59.5079710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5079818Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5080081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5080158Z return func(*args, **kwargs) 2025-10-10T00:57:59.5080428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5080529Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5080533Z 2025-10-10T00:57:59.5080644Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5080864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5080936Z res = mod(**inputs) 2025-10-10T00:57:59.5081224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5081306Z outputs = self.model.decoder( 2025-10-10T00:57:59.5081575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5081663Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5081899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5082011Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5082270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5082348Z return func(*args, **kwargs) 2025-10-10T00:57:59.5082640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5082770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5082774Z 2025-10-10T00:57:59.5082894Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5083105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5083202Z res = mod(**inputs) 2025-10-10T00:57:59.5083490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5083571Z outputs = self.model.decoder( 2025-10-10T00:57:59.5083853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5083932Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5084181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5084271Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5084527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5084611Z return func(*args, **kwargs) 2025-10-10T00:57:59.5084882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5085020Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5085248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5085329Z return self.act(input) 2025-10-10T00:57:59.5085341Z 2025-10-10T00:57:59.5085455Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5085674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5085754Z res = mod(**inputs) 2025-10-10T00:57:59.5086036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5086126Z outputs = self.model.decoder( 2025-10-10T00:57:59.5086406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5086492Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5086745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5086834Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5087108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5087186Z return func(*args, **kwargs) 2025-10-10T00:57:59.5087470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5087566Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5087570Z 2025-10-10T00:57:59.5087679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5087898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5087967Z res = mod(**inputs) 2025-10-10T00:57:59.5088241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5088326Z outputs = self.model.decoder( 2025-10-10T00:57:59.5088616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5088703Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5088939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5089048Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5089307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5089383Z return func(*args, **kwargs) 2025-10-10T00:57:59.5089658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.5089787Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.5089792Z 2025-10-10T00:57:59.5089910Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5090123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5090197Z res = mod(**inputs) 2025-10-10T00:57:59.5090476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5090557Z outputs = self.model.decoder( 2025-10-10T00:57:59.5090838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5090915Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5091161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5091248Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5091510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5091593Z return func(*args, **kwargs) 2025-10-10T00:57:59.5091866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5091982Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5092262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5092340Z return func(*args, **kwargs) 2025-10-10T00:57:59.5092627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5092805Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5092809Z 2025-10-10T00:57:59.5092927Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5093143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5093213Z res = mod(**inputs) 2025-10-10T00:57:59.5093493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5093575Z outputs = self.model.decoder( 2025-10-10T00:57:59.5093858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5093937Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5094183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5094270Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5094529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5094617Z return func(*args, **kwargs) 2025-10-10T00:57:59.5094885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5095019Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5095285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5095361Z return func(*args, **kwargs) 2025-10-10T00:57:59.5095668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5095760Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5095764Z 2025-10-10T00:57:59.5095879Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5096098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5096176Z res = mod(**inputs) 2025-10-10T00:57:59.5096537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5096618Z outputs = self.model.decoder( 2025-10-10T00:57:59.5096900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5096977Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5097223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5097313Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5097600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5097684Z return func(*args, **kwargs) 2025-10-10T00:57:59.5097974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5098091Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5098348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5098427Z return func(*args, **kwargs) 2025-10-10T00:57:59.5098701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5098794Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5098798Z 2025-10-10T00:57:59.5098897Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5099007Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5099226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5099296Z res = mod(**inputs) 2025-10-10T00:57:59.5099567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5099659Z outputs = self.model.decoder( 2025-10-10T00:57:59.5099932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5100018Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5100255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5100339Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5100611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5100687Z return func(*args, **kwargs) 2025-10-10T00:57:59.5100963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5101069Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5101327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5101410Z return func(*args, **kwargs) 2025-10-10T00:57:59.5101682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5101823Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5102137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5102312Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5102317Z 2025-10-10T00:57:59.5102430Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5102648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5102730Z res = mod(**inputs) 2025-10-10T00:57:59.5103030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5103137Z outputs = self.model.decoder( 2025-10-10T00:57:59.5103417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5103500Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5103750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5103838Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5104111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5104187Z return func(*args, **kwargs) 2025-10-10T00:57:59.5104471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5104581Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5104848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5104935Z return func(*args, **kwargs) 2025-10-10T00:57:59.5105212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5105310Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5105314Z 2025-10-10T00:57:59.5105427Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5105645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5105725Z res = mod(**inputs) 2025-10-10T00:57:59.5106004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5106093Z outputs = self.model.decoder( 2025-10-10T00:57:59.5106457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5106545Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5106799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5106888Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5107162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5107241Z return func(*args, **kwargs) 2025-10-10T00:57:59.5107526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5107659Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5107663Z 2025-10-10T00:57:59.5107776Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5108011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5108084Z res = mod(**inputs) 2025-10-10T00:57:59.5108374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5108481Z outputs = self.model.decoder( 2025-10-10T00:57:59.5108762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5108850Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5109115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5109216Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5109484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5109570Z return func(*args, **kwargs) 2025-10-10T00:57:59.5109889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5110022Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5110269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5110348Z return self.act(input) 2025-10-10T00:57:59.5110352Z 2025-10-10T00:57:59.5110475Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5110695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5110768Z res = mod(**inputs) 2025-10-10T00:57:59.5111060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5111141Z outputs = self.model.decoder( 2025-10-10T00:57:59.5111442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5111525Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5111778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5111876Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5112157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5112243Z return func(*args, **kwargs) 2025-10-10T00:57:59.5112520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5112617Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5112621Z 2025-10-10T00:57:59.5112732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5112952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5113033Z res = mod(**inputs) 2025-10-10T00:57:59.5113322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5113412Z outputs = self.model.decoder( 2025-10-10T00:57:59.5113701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5113780Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5114040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5114127Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5114402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5114479Z return func(*args, **kwargs) 2025-10-10T00:57:59.5114765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5114876Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5115149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5115254Z return func(*args, **kwargs) 2025-10-10T00:57:59.5115535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5115729Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5115733Z 2025-10-10T00:57:59.5115848Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5116067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5116147Z res = mod(**inputs) 2025-10-10T00:57:59.5116459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5116567Z outputs = self.model.decoder( 2025-10-10T00:57:59.5116858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5116939Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5117199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5117288Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5117564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5117642Z return func(*args, **kwargs) 2025-10-10T00:57:59.5117925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5118036Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5118303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5118389Z return func(*args, **kwargs) 2025-10-10T00:57:59.5118669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5118766Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5118770Z 2025-10-10T00:57:59.5118884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5119103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5119183Z res = mod(**inputs) 2025-10-10T00:57:59.5119462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5119547Z outputs = self.model.decoder( 2025-10-10T00:57:59.5119827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5119917Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5120161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5120249Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5120518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5120594Z return func(*args, **kwargs) 2025-10-10T00:57:59.5120880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5120987Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5121251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5121338Z return func(*args, **kwargs) 2025-10-10T00:57:59.5121616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5121757Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5121761Z 2025-10-10T00:57:59.5121850Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5121961Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5122187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5122276Z res = mod(**inputs) 2025-10-10T00:57:59.5122562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5122642Z outputs = self.model.decoder( 2025-10-10T00:57:59.5122927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5123044Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5123292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5123390Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5123657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5123741Z return func(*args, **kwargs) 2025-10-10T00:57:59.5124023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5124131Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5124407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5124485Z return func(*args, **kwargs) 2025-10-10T00:57:59.5124771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5124883Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5125205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5125364Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5125368Z 2025-10-10T00:57:59.5125483Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5125714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5125787Z res = mod(**inputs) 2025-10-10T00:57:59.5126076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5126160Z outputs = self.model.decoder( 2025-10-10T00:57:59.5126466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5126559Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5126807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5126905Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5127173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5127251Z return func(*args, **kwargs) 2025-10-10T00:57:59.5127540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5127651Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5127924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5128002Z return func(*args, **kwargs) 2025-10-10T00:57:59.5128290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5128383Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5128405Z 2025-10-10T00:57:59.5128521Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5128746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5128817Z res = mod(**inputs) 2025-10-10T00:57:59.5129126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5129209Z outputs = self.model.decoder( 2025-10-10T00:57:59.5129489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5129577Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5129841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5129954Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5130222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5130301Z return func(*args, **kwargs) 2025-10-10T00:57:59.5130588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5130719Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5130723Z 2025-10-10T00:57:59.5130846Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5131066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5131145Z res = mod(**inputs) 2025-10-10T00:57:59.5131428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5131675Z outputs = self.model.decoder( 2025-10-10T00:57:59.5131972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5132058Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5132312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5132401Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5132672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5132761Z return func(*args, **kwargs) 2025-10-10T00:57:59.5133040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5133178Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5133420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5133507Z return self.act(input) 2025-10-10T00:57:59.5133513Z 2025-10-10T00:57:59.5133629Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5133848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5133929Z res = mod(**inputs) 2025-10-10T00:57:59.5134212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5134301Z outputs = self.model.decoder( 2025-10-10T00:57:59.5134582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5134663Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5134921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5135012Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5135289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5135421Z return func(*args, **kwargs) 2025-10-10T00:57:59.5135700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5135799Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5135803Z 2025-10-10T00:57:59.5135943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5136171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5136243Z res = mod(**inputs) 2025-10-10T00:57:59.5136530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5136662Z outputs = self.model.decoder( 2025-10-10T00:57:59.5136943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5137037Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5137280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5137381Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5137649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5137726Z return func(*args, **kwargs) 2025-10-10T00:57:59.5138018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.5138103Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.5138106Z 2025-10-10T00:57:59.5138221Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5138422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5138497Z res = mod(**inputs) 2025-10-10T00:57:59.5138760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5138838Z outputs = self.model.decoder( 2025-10-10T00:57:59.5139117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5139197Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5139440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5139526Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5139783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5139870Z return func(*args, **kwargs) 2025-10-10T00:57:59.5140143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5140260Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5140519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5140593Z return func(*args, **kwargs) 2025-10-10T00:57:59.5140871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5141033Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5141037Z 2025-10-10T00:57:59.5141154Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5141368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5141449Z res = mod(**inputs) 2025-10-10T00:57:59.5141723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5141822Z outputs = self.model.decoder( 2025-10-10T00:57:59.5142101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5142179Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5142444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5142534Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5142796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5142882Z return func(*args, **kwargs) 2025-10-10T00:57:59.5143176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5143321Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5143586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5143672Z return func(*args, **kwargs) 2025-10-10T00:57:59.5143954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5144044Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5144050Z 2025-10-10T00:57:59.5144171Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5144387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5144466Z res = mod(**inputs) 2025-10-10T00:57:59.5144746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5144830Z outputs = self.model.decoder( 2025-10-10T00:57:59.5145127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5145209Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5145459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5145547Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5145814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5145898Z return func(*args, **kwargs) 2025-10-10T00:57:59.5146175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5146291Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5146621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5146705Z return func(*args, **kwargs) 2025-10-10T00:57:59.5146983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5147081Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5147085Z 2025-10-10T00:57:59.5147184Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5147301Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5147532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5147606Z res = mod(**inputs) 2025-10-10T00:57:59.5147890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5147982Z outputs = self.model.decoder( 2025-10-10T00:57:59.5148267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5148354Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5148588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5148696Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5148953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5149027Z return func(*args, **kwargs) 2025-10-10T00:57:59.5149316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5149426Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5149694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5149808Z return func(*args, **kwargs) 2025-10-10T00:57:59.5150078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5150194Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5150504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5150655Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5150659Z 2025-10-10T00:57:59.5150769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5150979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5151057Z res = mod(**inputs) 2025-10-10T00:57:59.5151326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5151414Z outputs = self.model.decoder( 2025-10-10T00:57:59.5151685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5151771Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5152015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5152097Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5152355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5152428Z return func(*args, **kwargs) 2025-10-10T00:57:59.5152694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5152798Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5153053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5153139Z return func(*args, **kwargs) 2025-10-10T00:57:59.5153417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5153512Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5153516Z 2025-10-10T00:57:59.5153620Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5153832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5153900Z res = mod(**inputs) 2025-10-10T00:57:59.5154167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5154253Z outputs = self.model.decoder( 2025-10-10T00:57:59.5154525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5154613Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5154849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5154954Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5155218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5155291Z return func(*args, **kwargs) 2025-10-10T00:57:59.5156590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5156738Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5156742Z 2025-10-10T00:57:59.5156854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5157073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5157166Z res = mod(**inputs) 2025-10-10T00:57:59.5157470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5157552Z outputs = self.model.decoder( 2025-10-10T00:57:59.5157838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5157918Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5158154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5158253Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5158518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5158602Z return func(*args, **kwargs) 2025-10-10T00:57:59.5158877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5159005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5159243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5159323Z return self.act(input) 2025-10-10T00:57:59.5159327Z 2025-10-10T00:57:59.5159444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5159659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5159734Z res = mod(**inputs) 2025-10-10T00:57:59.5160025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5160107Z outputs = self.model.decoder( 2025-10-10T00:57:59.5160399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5160481Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5160728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5160815Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5161071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5161154Z return func(*args, **kwargs) 2025-10-10T00:57:59.5161423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5161519Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5161523Z 2025-10-10T00:57:59.5161632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5161844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5161922Z res = mod(**inputs) 2025-10-10T00:57:59.5162194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5162281Z outputs = self.model.decoder( 2025-10-10T00:57:59.5162552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5162657Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5162899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5162985Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5163269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5163346Z return func(*args, **kwargs) 2025-10-10T00:57:59.5163626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5163769Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5164027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5164111Z return func(*args, **kwargs) 2025-10-10T00:57:59.5164383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-10-10T00:57:59.5164551Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T00:57:59.5164554Z 2025-10-10T00:57:59.5164666Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5164883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5164953Z res = mod(**inputs) 2025-10-10T00:57:59.5165228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5165317Z outputs = self.model.decoder( 2025-10-10T00:57:59.5165594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5165677Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5165917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5166003Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5166268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5166345Z return func(*args, **kwargs) 2025-10-10T00:57:59.5166624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5166731Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5166990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5167072Z return func(*args, **kwargs) 2025-10-10T00:57:59.5167341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-10-10T00:57:59.5167438Z key_states = self.k_proj(current_states) 2025-10-10T00:57:59.5167442Z 2025-10-10T00:57:59.5167549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5167767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5167837Z res = mod(**inputs) 2025-10-10T00:57:59.5168110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5168195Z outputs = self.model.decoder( 2025-10-10T00:57:59.5168465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5168554Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5168793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5168896Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5169164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5169238Z return func(*args, **kwargs) 2025-10-10T00:57:59.5169530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5169637Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5169892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5169972Z return func(*args, **kwargs) 2025-10-10T00:57:59.5170272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-10-10T00:57:59.5170391Z value_states = self.v_proj(current_states) 2025-10-10T00:57:59.5170395Z 2025-10-10T00:57:59.5170482Z cudagraph partition due to non gpu ops 2025-10-10T00:57:59.5170600Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5170814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5170883Z res = mod(**inputs) 2025-10-10T00:57:59.5171168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5171248Z outputs = self.model.decoder( 2025-10-10T00:57:59.5171529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5171606Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5171846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5171940Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5172199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5172283Z return func(*args, **kwargs) 2025-10-10T00:57:59.5172554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5172660Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5172930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5173003Z return func(*args, **kwargs) 2025-10-10T00:57:59.5173282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-10-10T00:57:59.5173391Z attn_output, attn_weights = attention_interface( 2025-10-10T00:57:59.5173714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T00:57:59.5173861Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T00:57:59.5173865Z 2025-10-10T00:57:59.5173975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5174195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5174265Z res = mod(**inputs) 2025-10-10T00:57:59.5174550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5174630Z outputs = self.model.decoder( 2025-10-10T00:57:59.5174907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5174994Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5175236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5175329Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5175613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5175697Z return func(*args, **kwargs) 2025-10-10T00:57:59.5175977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-10-10T00:57:59.5176101Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T00:57:59.5176381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5176459Z return func(*args, **kwargs) 2025-10-10T00:57:59.5176764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-10-10T00:57:59.5176884Z attn_output = self.out_proj(attn_output) 2025-10-10T00:57:59.5176887Z 2025-10-10T00:57:59.5176997Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5177218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5177289Z res = mod(**inputs) 2025-10-10T00:57:59.5177580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5177660Z outputs = self.model.decoder( 2025-10-10T00:57:59.5177941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5178020Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5178259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5178357Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5178614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5178700Z return func(*args, **kwargs) 2025-10-10T00:57:59.5178979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5179109Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5179113Z 2025-10-10T00:57:59.5179232Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5179452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5179533Z res = mod(**inputs) 2025-10-10T00:57:59.5179824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5179905Z outputs = self.model.decoder( 2025-10-10T00:57:59.5180207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5180286Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5180540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5180628Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5180913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5180991Z return func(*args, **kwargs) 2025-10-10T00:57:59.5181271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-10-10T00:57:59.5181409Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T00:57:59.5181646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:57:59.5181736Z return self.act(input) 2025-10-10T00:57:59.5181740Z 2025-10-10T00:57:59.5181853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5182073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5182176Z res = mod(**inputs) 2025-10-10T00:57:59.5182464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5182552Z outputs = self.model.decoder( 2025-10-10T00:57:59.5182873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5182956Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5183207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5183298Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5183606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5183687Z return func(*args, **kwargs) 2025-10-10T00:57:59.5183977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-10-10T00:57:59.5184072Z hidden_states = self.fc2(hidden_states) 2025-10-10T00:57:59.5184076Z 2025-10-10T00:57:59.5184190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5184419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5184493Z res = mod(**inputs) 2025-10-10T00:57:59.5184779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-10-10T00:57:59.5184858Z outputs = self.model.decoder( 2025-10-10T00:57:59.5185138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-10-10T00:57:59.5185228Z layer_outputs = decoder_layer( 2025-10-10T00:57:59.5185471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:57:59.5185568Z return super().__call__(*args, **kwargs) 2025-10-10T00:57:59.5185834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:57:59.5185917Z return func(*args, **kwargs) 2025-10-10T00:57:59.5186195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-10-10T00:57:59.5186286Z hidden_states = residual + hidden_states 2025-10-10T00:57:59.5186290Z 2025-10-10T00:57:59.5186657Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5186881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5186965Z res = mod(**inputs) 2025-10-10T00:57:59.5187246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1881, in forward 2025-10-10T00:57:59.5187337Z logits = self.lm_head(outputs[0]) 2025-10-10T00:57:59.5187341Z 2025-10-10T00:57:59.5187464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:57:59.5187680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:57:59.5187763Z res = mod(**inputs) 2025-10-10T00:57:59.5188047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1887, in forward 2025-10-10T00:57:59.5188214Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T00:57:59.5188226Z 2025-10-10T00:58:10.8595685Z Compilation time (from dynamo_timed): 17.333872059 2025-10-10T00:58:10.8879898Z pass 2025-10-10T00:58:10.8880554Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:10.8881917Z TIMING: _recursive_pre_grad_passes:0.00732 _recursive_joint_graph_passes:0.69149 _recursive_post_grad_passes:0.07198 async_compile.wait:0.88246 code_gen:9.98833 inductor_compile:11.33226 backend_compile:14.63581 gc:0.00028 entire_frame_compile:17.33387 total_wall_time:17.33387 2025-10-10T00:58:10.8883826Z STATS: call_* op count: 337 | FakeTensorMode.__torch_dispatch__:7484 | FakeTensor.__torch_dispatch__:4449 | ProxyTorchDispatchMode.__torch_dispatch__:2017 2025-10-10T00:58:10.8884740Z Dynamo produced 1 graphs covering 337 ops with 0 graph breaks (0 unique) 2025-10-10T00:58:13.6299852Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:58:13.6301649Z import pynvml # type: ignore[import] 2025-10-10T00:58:17.1471857Z 2025-10-10T00:58:19.5975974Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:58:19.5976268Z loading model: 0it [00:02, ?it/s] 2025-10-10T00:58:19.5989169Z cpu eval MT5ForConditionalGeneration 2025-10-10T00:58:20.2327422Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:20.5152208Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:20.7985176Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:33.4724357Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4727240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4727849Z res = mod(**inputs) 2025-10-10T00:58:33.4728465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.4728948Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.4729431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4729860Z layer_outputs = layer_module( 2025-10-10T00:58:33.4730266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4730700Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4731150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4731850Z return func(*args, **kwargs) 2025-10-10T00:58:33.4732252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4732686Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4733114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4733540Z return func(*args, **kwargs) 2025-10-10T00:58:33.4733919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4734313Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4734713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4735104Z return func(*args, **kwargs) 2025-10-10T00:58:33.4735466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 423, in forward 2025-10-10T00:58:33.4735847Z position_bias = position_bias + causal_mask 2025-10-10T00:58:33.4736003Z 2025-10-10T00:58:33.4736116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4736513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4736878Z res = mod(**inputs) 2025-10-10T00:58:33.4737264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4738023Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4738457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4738885Z layer_outputs = layer_module( 2025-10-10T00:58:33.4739323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4739702Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4740083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4740502Z return func(*args, **kwargs) 2025-10-10T00:58:33.4741890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4742328Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4742938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4743357Z return func(*args, **kwargs) 2025-10-10T00:58:33.4743775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4744223Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4744642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4745086Z return func(*args, **kwargs) 2025-10-10T00:58:33.4745499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.4745935Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.4746089Z 2025-10-10T00:58:33.4746220Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4746865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4747228Z res = mod(**inputs) 2025-10-10T00:58:33.4747628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4748035Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4748413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4748785Z layer_outputs = layer_module( 2025-10-10T00:58:33.4749142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4749508Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4749894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4750267Z return func(*args, **kwargs) 2025-10-10T00:58:33.4750628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4751054Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4751448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4751856Z return func(*args, **kwargs) 2025-10-10T00:58:33.4752250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4752676Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4753223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4753846Z return func(*args, **kwargs) 2025-10-10T00:58:33.4754300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.4754807Z key_states = self.k(current_states) 2025-10-10T00:58:33.4755003Z 2025-10-10T00:58:33.4755224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4755664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4756072Z res = mod(**inputs) 2025-10-10T00:58:33.4756586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4757068Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4757561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4758015Z layer_outputs = layer_module( 2025-10-10T00:58:33.4758512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4773727Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4774328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4774774Z return func(*args, **kwargs) 2025-10-10T00:58:33.4775204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4775665Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4776110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4776529Z return func(*args, **kwargs) 2025-10-10T00:58:33.4776941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4777391Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4777829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4778243Z return func(*args, **kwargs) 2025-10-10T00:58:33.4778650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.4779081Z value_states = self.v(current_states) 2025-10-10T00:58:33.4779237Z 2025-10-10T00:58:33.4779345Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4779586Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4779863Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4780286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4780652Z res = mod(**inputs) 2025-10-10T00:58:33.4781061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4781487Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4781918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4782348Z layer_outputs = layer_module( 2025-10-10T00:58:33.4782743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4783148Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4783585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4784012Z return func(*args, **kwargs) 2025-10-10T00:58:33.4784433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4784861Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4785292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4785707Z return func(*args, **kwargs) 2025-10-10T00:58:33.4786230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4786769Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4787208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4787663Z return func(*args, **kwargs) 2025-10-10T00:58:33.4788073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.4788506Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.4788654Z 2025-10-10T00:58:33.4788787Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4789228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4789619Z res = mod(**inputs) 2025-10-10T00:58:33.4790018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4790434Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4790845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4791247Z layer_outputs = layer_module( 2025-10-10T00:58:33.4791632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4792032Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4792445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4792856Z return func(*args, **kwargs) 2025-10-10T00:58:33.4793245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4793680Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4794111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4794579Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4795023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.4795476Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.4795652Z 2025-10-10T00:58:33.4795769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4796162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4796513Z res = mod(**inputs) 2025-10-10T00:58:33.4796889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4797299Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4797701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4798109Z layer_outputs = layer_module( 2025-10-10T00:58:33.4798487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4798882Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4799296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4799702Z return func(*args, **kwargs) 2025-10-10T00:58:33.4800110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4800552Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4801005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4801461Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4801931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.4802362Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.4802513Z 2025-10-10T00:58:33.4802634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4803061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4803423Z res = mod(**inputs) 2025-10-10T00:58:33.4803816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4804244Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4804705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4805122Z layer_outputs = layer_module( 2025-10-10T00:58:33.4805510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4805918Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4806334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4806749Z return func(*args, **kwargs) 2025-10-10T00:58:33.4807144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4807583Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4808025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4808496Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4808967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.4809410Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.4809571Z 2025-10-10T00:58:33.4809697Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4810105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4810465Z res = mod(**inputs) 2025-10-10T00:58:33.4810851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4811261Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4811669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4812076Z layer_outputs = layer_module( 2025-10-10T00:58:33.4812474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4812880Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4813305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4813727Z return func(*args, **kwargs) 2025-10-10T00:58:33.4814121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4814567Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4815002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4815481Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4815944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.4816350Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.4816508Z 2025-10-10T00:58:33.4816623Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4817248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4817607Z res = mod(**inputs) 2025-10-10T00:58:33.4817985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4818443Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4818866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4819296Z layer_outputs = layer_module( 2025-10-10T00:58:33.4819697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4820136Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4820562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4820983Z return func(*args, **kwargs) 2025-10-10T00:58:33.4821401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4821826Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4822248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4822663Z return func(*args, **kwargs) 2025-10-10T00:58:33.4823067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4823509Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4823930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4824344Z return func(*args, **kwargs) 2025-10-10T00:58:33.4824745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.4825170Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.4825323Z 2025-10-10T00:58:33.4825454Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4825851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4826217Z res = mod(**inputs) 2025-10-10T00:58:33.4826685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4827111Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4827528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4827947Z layer_outputs = layer_module( 2025-10-10T00:58:33.4828344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4828755Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4829182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4829587Z return func(*args, **kwargs) 2025-10-10T00:58:33.4829992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4830421Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4830849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4831266Z return func(*args, **kwargs) 2025-10-10T00:58:33.4831858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4832296Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4832723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4833244Z return func(*args, **kwargs) 2025-10-10T00:58:33.4833636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.4834056Z key_states = self.k(current_states) 2025-10-10T00:58:33.4834212Z 2025-10-10T00:58:33.4834365Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4834774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4835134Z res = mod(**inputs) 2025-10-10T00:58:33.4835521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4836021Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4836445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4836856Z layer_outputs = layer_module( 2025-10-10T00:58:33.4837239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4837637Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4838063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4838461Z return func(*args, **kwargs) 2025-10-10T00:58:33.4838847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4839262Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4839648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4840024Z return func(*args, **kwargs) 2025-10-10T00:58:33.4840386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4840772Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4841160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4841533Z return func(*args, **kwargs) 2025-10-10T00:58:33.4841920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.4842319Z value_states = self.v(current_states) 2025-10-10T00:58:33.4842473Z 2025-10-10T00:58:33.4842561Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4842796Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4843053Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4843438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4843774Z res = mod(**inputs) 2025-10-10T00:58:33.4844151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4844557Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4844956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4845326Z layer_outputs = layer_module( 2025-10-10T00:58:33.4845683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4846058Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4846465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4846856Z return func(*args, **kwargs) 2025-10-10T00:58:33.4847240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4847648Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4848100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4848475Z return func(*args, **kwargs) 2025-10-10T00:58:33.4848840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4849233Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4849642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4850039Z return func(*args, **kwargs) 2025-10-10T00:58:33.4850423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.4850867Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.4851017Z 2025-10-10T00:58:33.4851130Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4851517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4851867Z res = mod(**inputs) 2025-10-10T00:58:33.4852246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4852644Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4853045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4853449Z layer_outputs = layer_module( 2025-10-10T00:58:33.4853825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4854211Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4854623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4855024Z return func(*args, **kwargs) 2025-10-10T00:58:33.4855413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4855834Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4856247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4856698Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4857142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.4857575Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.4857740Z 2025-10-10T00:58:33.4857861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4858247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4858595Z res = mod(**inputs) 2025-10-10T00:58:33.4858976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4859380Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4859772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4860177Z layer_outputs = layer_module( 2025-10-10T00:58:33.4860551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4860948Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4861365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4861771Z return func(*args, **kwargs) 2025-10-10T00:58:33.4862169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4862622Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4863053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4863510Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4863974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.4864399Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.4864559Z 2025-10-10T00:58:33.4864679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4865077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4865447Z res = mod(**inputs) 2025-10-10T00:58:33.4865854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4866276Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4866794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4867217Z layer_outputs = layer_module( 2025-10-10T00:58:33.4867602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4868023Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4868438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4868843Z return func(*args, **kwargs) 2025-10-10T00:58:33.4869232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4869655Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4870047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4870461Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4870873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.4871268Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.4871423Z 2025-10-10T00:58:33.4871532Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4871901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4872230Z res = mod(**inputs) 2025-10-10T00:58:33.4872587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4872968Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4873360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4873767Z layer_outputs = layer_module( 2025-10-10T00:58:33.4874142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4874513Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4874894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4875273Z return func(*args, **kwargs) 2025-10-10T00:58:33.4875645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4876044Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4876435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4876860Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4877296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.4877731Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.4877869Z 2025-10-10T00:58:33.4877983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4878344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4878693Z res = mod(**inputs) 2025-10-10T00:58:33.4879058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4879466Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4879858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4880265Z layer_outputs = layer_module( 2025-10-10T00:58:33.4880624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4881009Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4881417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4881813Z return func(*args, **kwargs) 2025-10-10T00:58:33.4882208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4882623Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4883039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4883444Z return func(*args, **kwargs) 2025-10-10T00:58:33.4883832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4884253Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4884666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4885067Z return func(*args, **kwargs) 2025-10-10T00:58:33.4885447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.4885857Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.4886012Z 2025-10-10T00:58:33.4886128Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4886521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4886870Z res = mod(**inputs) 2025-10-10T00:58:33.4887240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4887653Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4888056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4888464Z layer_outputs = layer_module( 2025-10-10T00:58:33.4888841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4889230Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4889642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4890044Z return func(*args, **kwargs) 2025-10-10T00:58:33.4890432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4890834Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4891252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4891654Z return func(*args, **kwargs) 2025-10-10T00:58:33.4892044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4892481Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4892889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4893290Z return func(*args, **kwargs) 2025-10-10T00:58:33.4893723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.4894146Z key_states = self.k(current_states) 2025-10-10T00:58:33.4894303Z 2025-10-10T00:58:33.4894420Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4894829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4895232Z res = mod(**inputs) 2025-10-10T00:58:33.4895633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4896060Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4896466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4896871Z layer_outputs = layer_module( 2025-10-10T00:58:33.4897244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4897640Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4898048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4898448Z return func(*args, **kwargs) 2025-10-10T00:58:33.4898836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4899243Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4899655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4900055Z return func(*args, **kwargs) 2025-10-10T00:58:33.4900442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4900845Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4901258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4901658Z return func(*args, **kwargs) 2025-10-10T00:58:33.4902043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.4902462Z value_states = self.v(current_states) 2025-10-10T00:58:33.4902608Z 2025-10-10T00:58:33.4902696Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4902928Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4903188Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4903574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4903918Z res = mod(**inputs) 2025-10-10T00:58:33.4904307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4904734Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4905146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4905575Z layer_outputs = layer_module( 2025-10-10T00:58:33.4905949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4906431Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4906875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4907311Z return func(*args, **kwargs) 2025-10-10T00:58:33.4907704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4908127Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4908569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4908986Z return func(*args, **kwargs) 2025-10-10T00:58:33.4909387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4909805Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4910248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4910677Z return func(*args, **kwargs) 2025-10-10T00:58:33.4911074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.4911496Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.4911645Z 2025-10-10T00:58:33.4911762Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4912164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4912523Z res = mod(**inputs) 2025-10-10T00:58:33.4912915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4913328Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4913743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4914160Z layer_outputs = layer_module( 2025-10-10T00:58:33.4914551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4914961Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4915373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4915782Z return func(*args, **kwargs) 2025-10-10T00:58:33.4916185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4916625Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4917049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4917521Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4917993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.4918432Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.4918598Z 2025-10-10T00:58:33.4918722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4919102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4919451Z res = mod(**inputs) 2025-10-10T00:58:33.4919828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4920235Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4920637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4921033Z layer_outputs = layer_module( 2025-10-10T00:58:33.4921410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4921808Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4922219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4922631Z return func(*args, **kwargs) 2025-10-10T00:58:33.4923014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4923435Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4923866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4924314Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4924750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.4925165Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.4925339Z 2025-10-10T00:58:33.4925467Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4925858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4926212Z res = mod(**inputs) 2025-10-10T00:58:33.4926585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4926994Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4927398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4927819Z layer_outputs = layer_module( 2025-10-10T00:58:33.4928200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4928602Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4929027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4929443Z return func(*args, **kwargs) 2025-10-10T00:58:33.4929844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4930265Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4930700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4931158Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4931812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.4932254Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.4932411Z 2025-10-10T00:58:33.4932526Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4932937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4933306Z res = mod(**inputs) 2025-10-10T00:58:33.4933688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4934104Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4934517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4934929Z layer_outputs = layer_module( 2025-10-10T00:58:33.4935317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4935720Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4936133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4936541Z return func(*args, **kwargs) 2025-10-10T00:58:33.4936942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4937369Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4937791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4938299Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4938741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.4939190Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.4939338Z 2025-10-10T00:58:33.4939459Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4939839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4940192Z res = mod(**inputs) 2025-10-10T00:58:33.4940604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4941040Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4941441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4941839Z layer_outputs = layer_module( 2025-10-10T00:58:33.4942231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4942632Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4943051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4943458Z return func(*args, **kwargs) 2025-10-10T00:58:33.4943850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4944276Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4944705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4945129Z return func(*args, **kwargs) 2025-10-10T00:58:33.4945523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4945959Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4946455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4946891Z return func(*args, **kwargs) 2025-10-10T00:58:33.4947298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.4947716Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.4947876Z 2025-10-10T00:58:33.4947995Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4948400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4948765Z res = mod(**inputs) 2025-10-10T00:58:33.4949151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4949581Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4950002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4950421Z layer_outputs = layer_module( 2025-10-10T00:58:33.4950818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4951220Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4951643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4952055Z return func(*args, **kwargs) 2025-10-10T00:58:33.4952458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4952880Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4953325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4953706Z return func(*args, **kwargs) 2025-10-10T00:58:33.4954093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4954519Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4954922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4955324Z return func(*args, **kwargs) 2025-10-10T00:58:33.4955719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.4956143Z key_states = self.k(current_states) 2025-10-10T00:58:33.4956281Z 2025-10-10T00:58:33.4956396Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4956755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4957088Z res = mod(**inputs) 2025-10-10T00:58:33.4957442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4957827Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4958200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4958581Z layer_outputs = layer_module( 2025-10-10T00:58:33.4958934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4959304Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4959690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4960060Z return func(*args, **kwargs) 2025-10-10T00:58:33.4960425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4960834Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4961247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4961656Z return func(*args, **kwargs) 2025-10-10T00:58:33.4962038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4962447Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4962857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4963256Z return func(*args, **kwargs) 2025-10-10T00:58:33.4963634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.4964043Z value_states = self.v(current_states) 2025-10-10T00:58:33.4964197Z 2025-10-10T00:58:33.4964289Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4964530Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.4964796Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4965195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4965563Z res = mod(**inputs) 2025-10-10T00:58:33.4965943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4966350Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4966723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4967125Z layer_outputs = layer_module( 2025-10-10T00:58:33.4967501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4967935Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4968363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4968771Z return func(*args, **kwargs) 2025-10-10T00:58:33.4969175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4969563Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4969960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4970360Z return func(*args, **kwargs) 2025-10-10T00:58:33.4970758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.4971187Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.4971595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4971993Z return func(*args, **kwargs) 2025-10-10T00:58:33.4972368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.4972772Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.4972924Z 2025-10-10T00:58:33.4973039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4973427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4973770Z res = mod(**inputs) 2025-10-10T00:58:33.4974138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4974545Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4974943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4975343Z layer_outputs = layer_module( 2025-10-10T00:58:33.4975709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4976101Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4976505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4976899Z return func(*args, **kwargs) 2025-10-10T00:58:33.4977281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.4977683Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.4978091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4978489Z return func(*args, **kwargs) 2025-10-10T00:58:33.4978876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-10-10T00:58:33.4979339Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.4979529Z 2025-10-10T00:58:33.4979635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4980002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4980329Z res = mod(**inputs) 2025-10-10T00:58:33.4980682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4981062Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4981466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4981869Z layer_outputs = layer_module( 2025-10-10T00:58:33.4982244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4982665Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4983076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4983484Z return func(*args, **kwargs) 2025-10-10T00:58:33.4983902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4984331Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4984754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4985200Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4985683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.4986119Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.4986289Z 2025-10-10T00:58:33.4986496Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4986896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4987254Z res = mod(**inputs) 2025-10-10T00:58:33.4987652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4988062Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4988467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4988869Z layer_outputs = layer_module( 2025-10-10T00:58:33.4989228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4989602Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4989993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4990376Z return func(*args, **kwargs) 2025-10-10T00:58:33.4990739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4991137Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4991589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4992018Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4992432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.4992828Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.4992973Z 2025-10-10T00:58:33.4993079Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.4993447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.4993799Z res = mod(**inputs) 2025-10-10T00:58:33.4994168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.4994574Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.4994976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.4995393Z layer_outputs = layer_module( 2025-10-10T00:58:33.4995750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.4996116Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.4996507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.4996910Z return func(*args, **kwargs) 2025-10-10T00:58:33.4997331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.4997786Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.4998204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.4998675Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.4999126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.4999547Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.4999703Z 2025-10-10T00:58:33.4999817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5000255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5000611Z res = mod(**inputs) 2025-10-10T00:58:33.5000993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5001406Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5001801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5002210Z layer_outputs = layer_module( 2025-10-10T00:58:33.5002591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5002985Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5003388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5003793Z return func(*args, **kwargs) 2025-10-10T00:58:33.5004190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5004617Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5005041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5005481Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5005929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5006339Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5006484Z 2025-10-10T00:58:33.5006605Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5006995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5007339Z res = mod(**inputs) 2025-10-10T00:58:33.5007725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5008139Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5008541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5008957Z layer_outputs = layer_module( 2025-10-10T00:58:33.5009338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5009735Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5010145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5010560Z return func(*args, **kwargs) 2025-10-10T00:58:33.5010945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5011362Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5011777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5012195Z return func(*args, **kwargs) 2025-10-10T00:58:33.5012574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5012985Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5013412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5013808Z return func(*args, **kwargs) 2025-10-10T00:58:33.5014195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5014596Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5014747Z 2025-10-10T00:58:33.5014861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5015289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5015643Z res = mod(**inputs) 2025-10-10T00:58:33.5016029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5016430Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5016834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5017243Z layer_outputs = layer_module( 2025-10-10T00:58:33.5017623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5018013Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5018427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5018839Z return func(*args, **kwargs) 2025-10-10T00:58:33.5019238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5019658Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5020063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5020465Z return func(*args, **kwargs) 2025-10-10T00:58:33.5020860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5021279Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5021689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5022098Z return func(*args, **kwargs) 2025-10-10T00:58:33.5022491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5022907Z key_states = self.k(current_states) 2025-10-10T00:58:33.5023051Z 2025-10-10T00:58:33.5023171Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5023575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5023933Z res = mod(**inputs) 2025-10-10T00:58:33.5024327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5024753Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5025168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5025582Z layer_outputs = layer_module( 2025-10-10T00:58:33.5025964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5026443Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5026881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5027315Z return func(*args, **kwargs) 2025-10-10T00:58:33.5027724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5028153Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5028586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5028988Z return func(*args, **kwargs) 2025-10-10T00:58:33.5029371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5029789Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5030215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5030636Z return func(*args, **kwargs) 2025-10-10T00:58:33.5031026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5031446Z value_states = self.v(current_states) 2025-10-10T00:58:33.5031719Z 2025-10-10T00:58:33.5031811Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5032052Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5032314Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5032721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5033082Z res = mod(**inputs) 2025-10-10T00:58:33.5033478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5033904Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5034316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5034744Z layer_outputs = layer_module( 2025-10-10T00:58:33.5035134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5035543Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5035963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5036369Z return func(*args, **kwargs) 2025-10-10T00:58:33.5036767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5037191Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5037610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5038022Z return func(*args, **kwargs) 2025-10-10T00:58:33.5038421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5038850Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5039269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5039681Z return func(*args, **kwargs) 2025-10-10T00:58:33.5040075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5040500Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5040656Z 2025-10-10T00:58:33.5040769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5041170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5041520Z res = mod(**inputs) 2025-10-10T00:58:33.5041896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5042309Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5042717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5043186Z layer_outputs = layer_module( 2025-10-10T00:58:33.5043561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5043958Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5044390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5044792Z return func(*args, **kwargs) 2025-10-10T00:58:33.5045181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5045577Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5046037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5046465Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5046887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5047297Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5047459Z 2025-10-10T00:58:33.5047571Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5047965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5048307Z res = mod(**inputs) 2025-10-10T00:58:33.5048672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5049081Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5049474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5049856Z layer_outputs = layer_module( 2025-10-10T00:58:33.5050217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5050587Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5050967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5051348Z return func(*args, **kwargs) 2025-10-10T00:58:33.5051721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5052145Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5052561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5052980Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5053397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5053804Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5053949Z 2025-10-10T00:58:33.5054070Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5054458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5054781Z res = mod(**inputs) 2025-10-10T00:58:33.5055139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5055524Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5055903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5056278Z layer_outputs = layer_module( 2025-10-10T00:58:33.5056638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5057011Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5057429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5057807Z return func(*args, **kwargs) 2025-10-10T00:58:33.5058168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5058585Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5058978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5059399Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5059839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5060277Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5060438Z 2025-10-10T00:58:33.5060552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5060945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5061292Z res = mod(**inputs) 2025-10-10T00:58:33.5061662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5062070Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5062471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5062875Z layer_outputs = layer_module( 2025-10-10T00:58:33.5063245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5063632Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5064036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5064433Z return func(*args, **kwargs) 2025-10-10T00:58:33.5064820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5065238Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5065654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5066099Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5066637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5067074Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5067225Z 2025-10-10T00:58:33.5067349Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5067749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5068110Z res = mod(**inputs) 2025-10-10T00:58:33.5068472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5068847Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5069210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5069585Z layer_outputs = layer_module( 2025-10-10T00:58:33.5069933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5070302Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5070674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5071048Z return func(*args, **kwargs) 2025-10-10T00:58:33.5071407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5071820Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5072204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5072571Z return func(*args, **kwargs) 2025-10-10T00:58:33.5072954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5073336Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5073718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5074098Z return func(*args, **kwargs) 2025-10-10T00:58:33.5074476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5074894Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5075032Z 2025-10-10T00:58:33.5075151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5075525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5075856Z res = mod(**inputs) 2025-10-10T00:58:33.5076217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5076607Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5076986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5077370Z layer_outputs = layer_module( 2025-10-10T00:58:33.5077722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5078104Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5078494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5078878Z return func(*args, **kwargs) 2025-10-10T00:58:33.5079256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5079670Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5080085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5080487Z return func(*args, **kwargs) 2025-10-10T00:58:33.5080922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5081323Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5081720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5082101Z return func(*args, **kwargs) 2025-10-10T00:58:33.5082468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5082857Z key_states = self.k(current_states) 2025-10-10T00:58:33.5082994Z 2025-10-10T00:58:33.5083101Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5083470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5083797Z res = mod(**inputs) 2025-10-10T00:58:33.5084153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5084532Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5084915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5085303Z layer_outputs = layer_module( 2025-10-10T00:58:33.5085660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5086049Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5086426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5086807Z return func(*args, **kwargs) 2025-10-10T00:58:33.5087189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5087580Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5087971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5088362Z return func(*args, **kwargs) 2025-10-10T00:58:33.5088777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5089209Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5089621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5090016Z return func(*args, **kwargs) 2025-10-10T00:58:33.5090404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5090825Z value_states = self.v(current_states) 2025-10-10T00:58:33.5090974Z 2025-10-10T00:58:33.5091071Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5091304Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5091566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5091940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5092271Z res = mod(**inputs) 2025-10-10T00:58:33.5092634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5093013Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5093408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5093825Z layer_outputs = layer_module( 2025-10-10T00:58:33.5094200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5094577Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5094957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5095334Z return func(*args, **kwargs) 2025-10-10T00:58:33.5095701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5096096Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5096482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5096861Z return func(*args, **kwargs) 2025-10-10T00:58:33.5097233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5097629Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5098019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5098392Z return func(*args, **kwargs) 2025-10-10T00:58:33.5098763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5099153Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5099286Z 2025-10-10T00:58:33.5099405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5099807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5100155Z res = mod(**inputs) 2025-10-10T00:58:33.5100565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5100949Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5101331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5101728Z layer_outputs = layer_module( 2025-10-10T00:58:33.5102088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5102466Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5102851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5103272Z return func(*args, **kwargs) 2025-10-10T00:58:33.5103654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5104069Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5104478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5104885Z return func(*args, **kwargs) 2025-10-10T00:58:33.5105267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-10-10T00:58:33.5105734Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5105942Z 2025-10-10T00:58:33.5106056Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5106525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5106903Z res = mod(**inputs) 2025-10-10T00:58:33.5107292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5107715Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5108138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5108549Z layer_outputs = layer_module( 2025-10-10T00:58:33.5108931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5109332Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5109748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5110165Z return func(*args, **kwargs) 2025-10-10T00:58:33.5110555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5110993Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5111419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5111874Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5112320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5112762Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5112926Z 2025-10-10T00:58:33.5113039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5113431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5113788Z res = mod(**inputs) 2025-10-10T00:58:33.5114167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5114578Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5114975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5115416Z layer_outputs = layer_module( 2025-10-10T00:58:33.5115799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5116199Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5116637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5117048Z return func(*args, **kwargs) 2025-10-10T00:58:33.5117436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5117863Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5118303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5118765Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5119208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5119622Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5119768Z 2025-10-10T00:58:33.5119888Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5120279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5120625Z res = mod(**inputs) 2025-10-10T00:58:33.5121005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5121447Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5121846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5122250Z layer_outputs = layer_module( 2025-10-10T00:58:33.5122626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5123018Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5123427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5123827Z return func(*args, **kwargs) 2025-10-10T00:58:33.5124208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5124627Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5125045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5125492Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5125936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5126350Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5126513Z 2025-10-10T00:58:33.5126625Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5127014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5127359Z res = mod(**inputs) 2025-10-10T00:58:33.5127731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5128139Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5128539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5129011Z layer_outputs = layer_module( 2025-10-10T00:58:33.5129390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5129779Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5130186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5130613Z return func(*args, **kwargs) 2025-10-10T00:58:33.5131003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5131438Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5132021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5132438Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5132852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5133237Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5133429Z 2025-10-10T00:58:33.5133537Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5133899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5134227Z res = mod(**inputs) 2025-10-10T00:58:33.5134580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5134959Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5135323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5135697Z layer_outputs = layer_module( 2025-10-10T00:58:33.5136047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5136408Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5136798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5137205Z return func(*args, **kwargs) 2025-10-10T00:58:33.5137595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5138008Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5138418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5138805Z return func(*args, **kwargs) 2025-10-10T00:58:33.5139162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5139542Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5139922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5140299Z return func(*args, **kwargs) 2025-10-10T00:58:33.5140661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5141048Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5141192Z 2025-10-10T00:58:33.5141723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5142094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5142424Z res = mod(**inputs) 2025-10-10T00:58:33.5142781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5143171Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5143553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5143961Z layer_outputs = layer_module( 2025-10-10T00:58:33.5144332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5144730Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5145140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5145575Z return func(*args, **kwargs) 2025-10-10T00:58:33.5145963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5146433Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5146894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5147310Z return func(*args, **kwargs) 2025-10-10T00:58:33.5147715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5148146Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5148588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5148959Z return func(*args, **kwargs) 2025-10-10T00:58:33.5149317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5149746Z key_states = self.k(current_states) 2025-10-10T00:58:33.5149894Z 2025-10-10T00:58:33.5150011Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5150416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5150777Z res = mod(**inputs) 2025-10-10T00:58:33.5151163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5151248Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5151527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5151610Z layer_outputs = layer_module( 2025-10-10T00:58:33.5151858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5151959Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5152225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5152314Z return func(*args, **kwargs) 2025-10-10T00:58:33.5152580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5152672Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5152942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5153020Z return func(*args, **kwargs) 2025-10-10T00:58:33.5153293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5153386Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5153659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5153735Z return func(*args, **kwargs) 2025-10-10T00:58:33.5153996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5154094Z value_states = self.v(current_states) 2025-10-10T00:58:33.5154098Z 2025-10-10T00:58:33.5154190Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5154286Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5154400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5154621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5154705Z res = mod(**inputs) 2025-10-10T00:58:33.5154971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5155086Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5155352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5155434Z layer_outputs = layer_module( 2025-10-10T00:58:33.5155709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5155802Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5156075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5156151Z return func(*args, **kwargs) 2025-10-10T00:58:33.5156431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5156549Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5156817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5156903Z return func(*args, **kwargs) 2025-10-10T00:58:33.5157159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5157255Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5157515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5157590Z return func(*args, **kwargs) 2025-10-10T00:58:33.5157853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5157939Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5157944Z 2025-10-10T00:58:33.5158069Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5158285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5158358Z res = mod(**inputs) 2025-10-10T00:58:33.5158620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5158701Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5158967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5159046Z layer_outputs = layer_module( 2025-10-10T00:58:33.5159284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5159380Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5159637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5159721Z return func(*args, **kwargs) 2025-10-10T00:58:33.5159975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5160087Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5160345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5160472Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5160735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5160844Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5160848Z 2025-10-10T00:58:33.5160967Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5161183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5161255Z res = mod(**inputs) 2025-10-10T00:58:33.5161522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5161632Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5161905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5161981Z layer_outputs = layer_module( 2025-10-10T00:58:33.5162247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5162337Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5162596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5162681Z return func(*args, **kwargs) 2025-10-10T00:58:33.5162956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5163081Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5163335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5163462Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5163725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5163815Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5163819Z 2025-10-10T00:58:33.5163937Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5164147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5164217Z res = mod(**inputs) 2025-10-10T00:58:33.5164485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5164566Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5164832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5164912Z layer_outputs = layer_module( 2025-10-10T00:58:33.5165158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5165244Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5165503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5165588Z return func(*args, **kwargs) 2025-10-10T00:58:33.5165844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5165949Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5166209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5166333Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5166602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5166697Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5166701Z 2025-10-10T00:58:33.5166821Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5167034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5167112Z res = mod(**inputs) 2025-10-10T00:58:33.5167378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5167452Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5167699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5167772Z layer_outputs = layer_module( 2025-10-10T00:58:33.5167997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5168098Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5168332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5168409Z return func(*args, **kwargs) 2025-10-10T00:58:33.5168659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5168758Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5168994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5169139Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5169387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5169470Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5169475Z 2025-10-10T00:58:33.5169587Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5169784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5169857Z res = mod(**inputs) 2025-10-10T00:58:33.5170104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5170179Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5170432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5170505Z layer_outputs = layer_module( 2025-10-10T00:58:33.5170740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5170822Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5171075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5171153Z return func(*args, **kwargs) 2025-10-10T00:58:33.5171389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5171479Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5171715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5171791Z return func(*args, **kwargs) 2025-10-10T00:58:33.5172024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5172109Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5172355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5172426Z return func(*args, **kwargs) 2025-10-10T00:58:33.5172669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5172749Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5172752Z 2025-10-10T00:58:33.5172856Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5173065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5173133Z res = mod(**inputs) 2025-10-10T00:58:33.5173390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5173466Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5173714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5173794Z layer_outputs = layer_module( 2025-10-10T00:58:33.5174050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5174140Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5174384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5174480Z return func(*args, **kwargs) 2025-10-10T00:58:33.5174725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5174809Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5175060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5175166Z return func(*args, **kwargs) 2025-10-10T00:58:33.5175430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5175523Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5175779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5175861Z return func(*args, **kwargs) 2025-10-10T00:58:33.5176119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5176210Z key_states = self.k(current_states) 2025-10-10T00:58:33.5176215Z 2025-10-10T00:58:33.5176325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5176548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5176627Z res = mod(**inputs) 2025-10-10T00:58:33.5176889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5176976Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5177251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5177331Z layer_outputs = layer_module( 2025-10-10T00:58:33.5177558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5177642Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5177901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5177975Z return func(*args, **kwargs) 2025-10-10T00:58:33.5178248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5178339Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5178599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5178679Z return func(*args, **kwargs) 2025-10-10T00:58:33.5178920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5179012Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5179258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5179327Z return func(*args, **kwargs) 2025-10-10T00:58:33.5179615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5179700Z value_states = self.v(current_states) 2025-10-10T00:58:33.5179704Z 2025-10-10T00:58:33.5179799Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5179888Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5180007Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5180228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5180322Z res = mod(**inputs) 2025-10-10T00:58:33.5180590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5180668Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5180953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5181035Z layer_outputs = layer_module( 2025-10-10T00:58:33.5181280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5181374Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5181661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5181772Z return func(*args, **kwargs) 2025-10-10T00:58:33.5182037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5182127Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5182398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5182472Z return func(*args, **kwargs) 2025-10-10T00:58:33.5182738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5182828Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5183098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5183174Z return func(*args, **kwargs) 2025-10-10T00:58:33.5183435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5183527Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5183532Z 2025-10-10T00:58:33.5183644Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5183865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5183935Z res = mod(**inputs) 2025-10-10T00:58:33.5184195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5184280Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5184538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5184621Z layer_outputs = layer_module( 2025-10-10T00:58:33.5184858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5184947Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5185216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5185294Z return func(*args, **kwargs) 2025-10-10T00:58:33.5185559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5185647Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5185917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5185991Z return func(*args, **kwargs) 2025-10-10T00:58:33.5186254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5186422Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5186694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5186776Z return func(*args, **kwargs) 2025-10-10T00:58:33.5187107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5187191Z key_states = self.k(current_states) 2025-10-10T00:58:33.5187196Z 2025-10-10T00:58:33.5187321Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5187553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5187636Z res = mod(**inputs) 2025-10-10T00:58:33.5187902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5187985Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5188276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5188382Z layer_outputs = layer_module( 2025-10-10T00:58:33.5188629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5188718Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5188985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5189060Z return func(*args, **kwargs) 2025-10-10T00:58:33.5189319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5189416Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5189672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5189754Z return func(*args, **kwargs) 2025-10-10T00:58:33.5190014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5190103Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5190369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5190445Z return func(*args, **kwargs) 2025-10-10T00:58:33.5190710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5190796Z value_states = self.v(current_states) 2025-10-10T00:58:33.5190800Z 2025-10-10T00:58:33.5190892Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5190978Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5191089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5191308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5191380Z res = mod(**inputs) 2025-10-10T00:58:33.5191649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5191730Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5191991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5192078Z layer_outputs = layer_module( 2025-10-10T00:58:33.5192317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5192412Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5192669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5192743Z return func(*args, **kwargs) 2025-10-10T00:58:33.5193008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5193096Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5193361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5193456Z return func(*args, **kwargs) 2025-10-10T00:58:33.5193708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5193806Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5194081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5194165Z return func(*args, **kwargs) 2025-10-10T00:58:33.5194423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5194513Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5194550Z 2025-10-10T00:58:33.5194665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5194875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5194955Z res = mod(**inputs) 2025-10-10T00:58:33.5195217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5195303Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5195564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5195643Z layer_outputs = layer_module( 2025-10-10T00:58:33.5195890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5195978Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5196246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5196322Z return func(*args, **kwargs) 2025-10-10T00:58:33.5196579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5196677Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5196933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5197015Z return func(*args, **kwargs) 2025-10-10T00:58:33.5197271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5197375Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5197637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5197705Z return func(*args, **kwargs) 2025-10-10T00:58:33.5197955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5198036Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5198041Z 2025-10-10T00:58:33.5198154Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5198352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5198416Z res = mod(**inputs) 2025-10-10T00:58:33.5198672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5198746Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5198996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5199069Z layer_outputs = layer_module( 2025-10-10T00:58:33.5199295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5199384Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5199625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5199724Z return func(*args, **kwargs) 2025-10-10T00:58:33.5199968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5200054Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5200318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5200389Z return func(*args, **kwargs) 2025-10-10T00:58:33.5200636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5200721Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5201002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5201073Z return func(*args, **kwargs) 2025-10-10T00:58:33.5201320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5201408Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5201412Z 2025-10-10T00:58:33.5201517Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5201727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5201793Z res = mod(**inputs) 2025-10-10T00:58:33.5202040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5202125Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5202377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5202460Z layer_outputs = layer_module( 2025-10-10T00:58:33.5202687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5202780Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5203027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5203096Z return func(*args, **kwargs) 2025-10-10T00:58:33.5203353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5203436Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5203689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5203761Z return func(*args, **kwargs) 2025-10-10T00:58:33.5204013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-10-10T00:58:33.5204158Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5204163Z 2025-10-10T00:58:33.5204272Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5204482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5204548Z res = mod(**inputs) 2025-10-10T00:58:33.5204806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5204882Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5205132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5205214Z layer_outputs = layer_module( 2025-10-10T00:58:33.5205449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5205546Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5205808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5205903Z return func(*args, **kwargs) 2025-10-10T00:58:33.5206173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5206274Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5206561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5206691Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5206947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5207115Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5207120Z 2025-10-10T00:58:33.5207231Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5207460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5207529Z res = mod(**inputs) 2025-10-10T00:58:33.5207781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5207857Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5208102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5208184Z layer_outputs = layer_module( 2025-10-10T00:58:33.5208407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5208497Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5208744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5208817Z return func(*args, **kwargs) 2025-10-10T00:58:33.5209069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5209165Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5209413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5209531Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5209781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5209864Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5209868Z 2025-10-10T00:58:33.5209972Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5210186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5210254Z res = mod(**inputs) 2025-10-10T00:58:33.5210507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5210586Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5210831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5210915Z layer_outputs = layer_module( 2025-10-10T00:58:33.5211141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5211230Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5211473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5211546Z return func(*args, **kwargs) 2025-10-10T00:58:33.5211809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5211908Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5212188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5212309Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5212584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5212684Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5212687Z 2025-10-10T00:58:33.5212800Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5213022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5213095Z res = mod(**inputs) 2025-10-10T00:58:33.5213400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-10-10T00:58:33.5213480Z encoder_outputs = self.encoder( 2025-10-10T00:58:33.5213739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5213826Z layer_outputs = layer_module( 2025-10-10T00:58:33.5214064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5214157Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5214416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5214490Z return func(*args, **kwargs) 2025-10-10T00:58:33.5214753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5214863Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5215115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5215231Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5215486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5215574Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5215578Z 2025-10-10T00:58:33.5215690Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5215912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5215982Z res = mod(**inputs) 2025-10-10T00:58:33.5216251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5216331Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5216594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5216678Z layer_outputs = layer_module( 2025-10-10T00:58:33.5216918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5217012Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5217273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5217357Z return func(*args, **kwargs) 2025-10-10T00:58:33.5217615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5217705Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5217971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5218051Z return func(*args, **kwargs) 2025-10-10T00:58:33.5218317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5218432Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5218691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5218773Z return func(*args, **kwargs) 2025-10-10T00:58:33.5219051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5219144Z key_states = self.k(current_states) 2025-10-10T00:58:33.5219148Z 2025-10-10T00:58:33.5219258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5219467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5219578Z res = mod(**inputs) 2025-10-10T00:58:33.5219883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5219972Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5220232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5220319Z layer_outputs = layer_module( 2025-10-10T00:58:33.5220557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5220645Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5220909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5220985Z return func(*args, **kwargs) 2025-10-10T00:58:33.5221248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5221341Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5221596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5221681Z return func(*args, **kwargs) 2025-10-10T00:58:33.5221936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5222038Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5222294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5222368Z return func(*args, **kwargs) 2025-10-10T00:58:33.5222629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5222715Z value_states = self.v(current_states) 2025-10-10T00:58:33.5222719Z 2025-10-10T00:58:33.5222821Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5222910Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5223032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5223248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5223320Z res = mod(**inputs) 2025-10-10T00:58:33.5223587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5223666Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5223933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5224012Z layer_outputs = layer_module( 2025-10-10T00:58:33.5224249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5224345Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5224605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5224687Z return func(*args, **kwargs) 2025-10-10T00:58:33.5224961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5225048Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5225309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5225401Z return func(*args, **kwargs) 2025-10-10T00:58:33.5225663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5225755Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5226017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5226136Z return func(*args, **kwargs) 2025-10-10T00:58:33.5226486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5226593Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5226597Z 2025-10-10T00:58:33.5226713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5226943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5227016Z res = mod(**inputs) 2025-10-10T00:58:33.5227289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5227378Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5227648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5227736Z layer_outputs = layer_module( 2025-10-10T00:58:33.5227986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5228074Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5228349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5228426Z return func(*args, **kwargs) 2025-10-10T00:58:33.5228707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5228805Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5229072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5229198Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5229457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5229575Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5229579Z 2025-10-10T00:58:33.5229689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5229909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5229978Z res = mod(**inputs) 2025-10-10T00:58:33.5230239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5230328Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5230588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5230674Z layer_outputs = layer_module( 2025-10-10T00:58:33.5230911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5231005Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5231263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5231366Z return func(*args, **kwargs) 2025-10-10T00:58:33.5231774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5231877Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5232903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5233040Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5233297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5233392Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5233396Z 2025-10-10T00:58:33.5233549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5233773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5233839Z res = mod(**inputs) 2025-10-10T00:58:33.5234084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5234167Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5234413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5234494Z layer_outputs = layer_module( 2025-10-10T00:58:33.5234711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5234797Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5235043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5235121Z return func(*args, **kwargs) 2025-10-10T00:58:33.5235382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5235480Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5235740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5235862Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5236118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5236221Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5236225Z 2025-10-10T00:58:33.5236336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5236555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5236630Z res = mod(**inputs) 2025-10-10T00:58:33.5236895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5236981Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5237235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5237322Z layer_outputs = layer_module( 2025-10-10T00:58:33.5237554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5237641Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5237876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5237945Z return func(*args, **kwargs) 2025-10-10T00:58:33.5238184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5238277Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5238517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5238690Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5238923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5239012Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5239016Z 2025-10-10T00:58:33.5239143Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5239347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5239413Z res = mod(**inputs) 2025-10-10T00:58:33.5239660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5239781Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5240021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5240104Z layer_outputs = layer_module( 2025-10-10T00:58:33.5240324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5240410Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5240648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5240717Z return func(*args, **kwargs) 2025-10-10T00:58:33.5240960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5241043Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5241285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5241357Z return func(*args, **kwargs) 2025-10-10T00:58:33.5241591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5241685Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5241923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5242002Z return func(*args, **kwargs) 2025-10-10T00:58:33.5242244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5242333Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5242337Z 2025-10-10T00:58:33.5242454Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5242649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5242726Z res = mod(**inputs) 2025-10-10T00:58:33.5242963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5243043Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5243278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5243349Z layer_outputs = layer_module( 2025-10-10T00:58:33.5243577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5243656Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5243899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5243966Z return func(*args, **kwargs) 2025-10-10T00:58:33.5244208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5244290Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5244526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5244622Z return func(*args, **kwargs) 2025-10-10T00:58:33.5244861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5244954Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5245214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5245285Z return func(*args, **kwargs) 2025-10-10T00:58:33.5245531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5245615Z key_states = self.k(current_states) 2025-10-10T00:58:33.5245636Z 2025-10-10T00:58:33.5245775Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5245989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5246060Z res = mod(**inputs) 2025-10-10T00:58:33.5246330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5246408Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5246674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5246752Z layer_outputs = layer_module( 2025-10-10T00:58:33.5246998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5247086Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5247341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5247425Z return func(*args, **kwargs) 2025-10-10T00:58:33.5247675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5247764Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5247999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5248068Z return func(*args, **kwargs) 2025-10-10T00:58:33.5248310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5248393Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5248635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5248705Z return func(*args, **kwargs) 2025-10-10T00:58:33.5248942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5249031Z value_states = self.v(current_states) 2025-10-10T00:58:33.5249036Z 2025-10-10T00:58:33.5249118Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5249206Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5249309Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5249518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5249586Z res = mod(**inputs) 2025-10-10T00:58:33.5249832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5249915Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5250160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5250245Z layer_outputs = layer_module( 2025-10-10T00:58:33.5250470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5250574Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5250823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5250893Z return func(*args, **kwargs) 2025-10-10T00:58:33.5251165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5251251Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5251491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5251568Z return func(*args, **kwargs) 2025-10-10T00:58:33.5251823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5251932Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5252172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5252250Z return func(*args, **kwargs) 2025-10-10T00:58:33.5252489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5252568Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5252571Z 2025-10-10T00:58:33.5252683Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5252883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5252954Z res = mod(**inputs) 2025-10-10T00:58:33.5253203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5253284Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5253549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5253627Z layer_outputs = layer_module( 2025-10-10T00:58:33.5253881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5253963Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5254204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5254280Z return func(*args, **kwargs) 2025-10-10T00:58:33.5254519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5254610Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5254850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5254928Z return func(*args, **kwargs) 2025-10-10T00:58:33.5255169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5255259Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5255510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5255579Z return func(*args, **kwargs) 2025-10-10T00:58:33.5255827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5255908Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5255912Z 2025-10-10T00:58:33.5256016Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5256222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5256292Z res = mod(**inputs) 2025-10-10T00:58:33.5256543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5256650Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5256893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5256975Z layer_outputs = layer_module( 2025-10-10T00:58:33.5257216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5257307Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5257549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5257629Z return func(*args, **kwargs) 2025-10-10T00:58:33.5257885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5257986Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5258236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5258320Z return func(*args, **kwargs) 2025-10-10T00:58:33.5258567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5258653Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5258897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5258977Z return func(*args, **kwargs) 2025-10-10T00:58:33.5259216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5259302Z key_states = self.k(current_states) 2025-10-10T00:58:33.5259307Z 2025-10-10T00:58:33.5259412Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5259615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5259688Z res = mod(**inputs) 2025-10-10T00:58:33.5259934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5260015Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5260260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5260338Z layer_outputs = layer_module( 2025-10-10T00:58:33.5260560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5260640Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5260890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5260961Z return func(*args, **kwargs) 2025-10-10T00:58:33.5261209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5261296Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5261536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5261613Z return func(*args, **kwargs) 2025-10-10T00:58:33.5261857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5261948Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5262188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5262258Z return func(*args, **kwargs) 2025-10-10T00:58:33.5262507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5262590Z value_states = self.v(current_states) 2025-10-10T00:58:33.5262613Z 2025-10-10T00:58:33.5262707Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5262790Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5262903Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5263103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5263170Z res = mod(**inputs) 2025-10-10T00:58:33.5263440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5263518Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5263786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5263882Z layer_outputs = layer_module( 2025-10-10T00:58:33.5264138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5264235Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5264492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5264576Z return func(*args, **kwargs) 2025-10-10T00:58:33.5264837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5264923Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5265173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5265246Z return func(*args, **kwargs) 2025-10-10T00:58:33.5265493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5265582Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5265831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5265903Z return func(*args, **kwargs) 2025-10-10T00:58:33.5266141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5266229Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5266232Z 2025-10-10T00:58:33.5266413Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5266633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5266708Z res = mod(**inputs) 2025-10-10T00:58:33.5266978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5267070Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5267339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5267429Z layer_outputs = layer_module( 2025-10-10T00:58:33.5267677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5267764Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5268037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5268116Z return func(*args, **kwargs) 2025-10-10T00:58:33.5268383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5268477Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5268730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5268863Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5269125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5269277Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5269281Z 2025-10-10T00:58:33.5269394Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5269617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5269708Z res = mod(**inputs) 2025-10-10T00:58:33.5269975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5270065Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5270334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5270460Z layer_outputs = layer_module( 2025-10-10T00:58:33.5270706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5270802Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5271066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5271143Z return func(*args, **kwargs) 2025-10-10T00:58:33.5271410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5271512Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5271778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5271906Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5272169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5272268Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5272271Z 2025-10-10T00:58:33.5272386Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5272609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5272681Z res = mod(**inputs) 2025-10-10T00:58:33.5272943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5273034Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5273297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5273382Z layer_outputs = layer_module( 2025-10-10T00:58:33.5273626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5273724Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5273986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5274063Z return func(*args, **kwargs) 2025-10-10T00:58:33.5274329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5274428Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5274697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5274825Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5275086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5275190Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5275195Z 2025-10-10T00:58:33.5275311Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5275534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5275636Z res = mod(**inputs) 2025-10-10T00:58:33.5275912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5275994Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5276274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5276365Z layer_outputs = layer_module( 2025-10-10T00:58:33.5276609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5276703Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5276964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5277058Z return func(*args, **kwargs) 2025-10-10T00:58:33.5277304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5277395Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5277644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5277759Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5278008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5278098Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5278102Z 2025-10-10T00:58:33.5278206Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5278418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5278486Z res = mod(**inputs) 2025-10-10T00:58:33.5278742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5278817Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5279066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5279150Z layer_outputs = layer_module( 2025-10-10T00:58:33.5279381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5279471Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5279720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5279792Z return func(*args, **kwargs) 2025-10-10T00:58:33.5280048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5280133Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5280389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5280466Z return func(*args, **kwargs) 2025-10-10T00:58:33.5280713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5280807Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5281059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5281137Z return func(*args, **kwargs) 2025-10-10T00:58:33.5281387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5281474Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5281481Z 2025-10-10T00:58:33.5281584Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5281782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5281875Z res = mod(**inputs) 2025-10-10T00:58:33.5282110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5282190Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5282446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5282521Z layer_outputs = layer_module( 2025-10-10T00:58:33.5282750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5282832Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5283095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5283183Z return func(*args, **kwargs) 2025-10-10T00:58:33.5283421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5283513Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5283753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5283831Z return func(*args, **kwargs) 2025-10-10T00:58:33.5284071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5284164Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5284413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5284481Z return func(*args, **kwargs) 2025-10-10T00:58:33.5284726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5284802Z key_states = self.k(current_states) 2025-10-10T00:58:33.5284808Z 2025-10-10T00:58:33.5284919Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5285113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5285179Z res = mod(**inputs) 2025-10-10T00:58:33.5285428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5285500Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5285747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5285821Z layer_outputs = layer_module( 2025-10-10T00:58:33.5286055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5286138Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5286387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5286466Z return func(*args, **kwargs) 2025-10-10T00:58:33.5286698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5286786Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5287019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5287088Z return func(*args, **kwargs) 2025-10-10T00:58:33.5287328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5287410Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5287659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5287729Z return func(*args, **kwargs) 2025-10-10T00:58:33.5287985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5288071Z value_states = self.v(current_states) 2025-10-10T00:58:33.5288074Z 2025-10-10T00:58:33.5288155Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5288240Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5288358Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5288554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5288623Z res = mod(**inputs) 2025-10-10T00:58:33.5288866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5288981Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5289224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5289314Z layer_outputs = layer_module( 2025-10-10T00:58:33.5289531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5289611Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5289853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5289923Z return func(*args, **kwargs) 2025-10-10T00:58:33.5290164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5290244Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5290479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5290562Z return func(*args, **kwargs) 2025-10-10T00:58:33.5290794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5290885Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5291118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5291186Z return func(*args, **kwargs) 2025-10-10T00:58:33.5291428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5291505Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5291509Z 2025-10-10T00:58:33.5291619Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5291816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5291891Z res = mod(**inputs) 2025-10-10T00:58:33.5292134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5292211Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5292460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5292534Z layer_outputs = layer_module( 2025-10-10T00:58:33.5292767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5292851Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5293092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5293171Z return func(*args, **kwargs) 2025-10-10T00:58:33.5293409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5293502Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5293753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5293844Z return func(*args, **kwargs) 2025-10-10T00:58:33.5294084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-10-10T00:58:33.5294218Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5294239Z 2025-10-10T00:58:33.5294361Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5294550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5294621Z res = mod(**inputs) 2025-10-10T00:58:33.5294854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5294962Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5295207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5295279Z layer_outputs = layer_module( 2025-10-10T00:58:33.5295502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5295580Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5295818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5295893Z return func(*args, **kwargs) 2025-10-10T00:58:33.5296127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5296218Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5296453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5296531Z return func(*args, **kwargs) 2025-10-10T00:58:33.5296774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5296862Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5297106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5297174Z return func(*args, **kwargs) 2025-10-10T00:58:33.5297422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5297498Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5297501Z 2025-10-10T00:58:33.5297602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5297800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5297868Z res = mod(**inputs) 2025-10-10T00:58:33.5298119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5298218Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5298463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5298551Z layer_outputs = layer_module( 2025-10-10T00:58:33.5298770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5298855Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5299091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5299168Z return func(*args, **kwargs) 2025-10-10T00:58:33.5299403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5299487Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5299731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5299817Z return func(*args, **kwargs) 2025-10-10T00:58:33.5300055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5300140Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5300394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5300473Z return func(*args, **kwargs) 2025-10-10T00:58:33.5300718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5300803Z key_states = self.k(current_states) 2025-10-10T00:58:33.5300846Z 2025-10-10T00:58:33.5300953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5301159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5301227Z res = mod(**inputs) 2025-10-10T00:58:33.5301470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5301553Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5301797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5301879Z layer_outputs = layer_module( 2025-10-10T00:58:33.5302110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5302194Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5302456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5302529Z return func(*args, **kwargs) 2025-10-10T00:58:33.5302790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5302881Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5303136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5303217Z return func(*args, **kwargs) 2025-10-10T00:58:33.5303472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5303572Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5303825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5303898Z return func(*args, **kwargs) 2025-10-10T00:58:33.5304161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5304245Z value_states = self.v(current_states) 2025-10-10T00:58:33.5304251Z 2025-10-10T00:58:33.5304343Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5304429Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5304545Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5304753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5304824Z res = mod(**inputs) 2025-10-10T00:58:33.5305089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5305167Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5305430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5305512Z layer_outputs = layer_module( 2025-10-10T00:58:33.5305748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5305859Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5306115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5306196Z return func(*args, **kwargs) 2025-10-10T00:58:33.5306559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5306658Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5306932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5307009Z return func(*args, **kwargs) 2025-10-10T00:58:33.5307298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5307413Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5307688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5307768Z return func(*args, **kwargs) 2025-10-10T00:58:33.5308033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5308128Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5308133Z 2025-10-10T00:58:33.5308251Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5308482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5308565Z res = mod(**inputs) 2025-10-10T00:58:33.5308827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5308920Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5309189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5309274Z layer_outputs = layer_module( 2025-10-10T00:58:33.5309502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5309582Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5309856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5309933Z return func(*args, **kwargs) 2025-10-10T00:58:33.5310209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5310312Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5310586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5310719Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5310982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5311105Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5311109Z 2025-10-10T00:58:33.5311222Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5311452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5311525Z res = mod(**inputs) 2025-10-10T00:58:33.5311796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5311884Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5312154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5312241Z layer_outputs = layer_module( 2025-10-10T00:58:33.5312488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5312604Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5312869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5312945Z return func(*args, **kwargs) 2025-10-10T00:58:33.5313234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5313337Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5313610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5313737Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5314034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5314132Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5314138Z 2025-10-10T00:58:33.5314250Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5314477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5314548Z res = mod(**inputs) 2025-10-10T00:58:33.5314814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5314903Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5315168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5315256Z layer_outputs = layer_module( 2025-10-10T00:58:33.5315501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5315600Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5315863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5315941Z return func(*args, **kwargs) 2025-10-10T00:58:33.5316213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5316313Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5316582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5316711Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5316973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5317084Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5317088Z 2025-10-10T00:58:33.5317202Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5317424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5317502Z res = mod(**inputs) 2025-10-10T00:58:33.5317773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5317854Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5318119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5318205Z layer_outputs = layer_module( 2025-10-10T00:58:33.5318448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5318543Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5318810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5318886Z return func(*args, **kwargs) 2025-10-10T00:58:33.5319153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5319273Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5319541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5319707Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5319975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5320071Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5320074Z 2025-10-10T00:58:33.5320184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5320440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5320512Z res = mod(**inputs) 2025-10-10T00:58:33.5320775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5320855Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5321111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5321196Z layer_outputs = layer_module( 2025-10-10T00:58:33.5321432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5321526Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5321780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5321856Z return func(*args, **kwargs) 2025-10-10T00:58:33.5322120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5322206Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5322469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5322545Z return func(*args, **kwargs) 2025-10-10T00:58:33.5322799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5322897Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5323150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5323230Z return func(*args, **kwargs) 2025-10-10T00:58:33.5323484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5323578Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5323582Z 2025-10-10T00:58:33.5323692Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5323902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5323980Z res = mod(**inputs) 2025-10-10T00:58:33.5324238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5324323Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5324582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5324660Z layer_outputs = layer_module( 2025-10-10T00:58:33.5324904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5324988Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5325253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5325327Z return func(*args, **kwargs) 2025-10-10T00:58:33.5325611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5325699Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5325957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5326063Z return func(*args, **kwargs) 2025-10-10T00:58:33.5326321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5326419Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5326679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5326787Z return func(*args, **kwargs) 2025-10-10T00:58:33.5327049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5327133Z key_states = self.k(current_states) 2025-10-10T00:58:33.5327137Z 2025-10-10T00:58:33.5327256Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5327468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5327541Z res = mod(**inputs) 2025-10-10T00:58:33.5327815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5327896Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5328169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5328248Z layer_outputs = layer_module( 2025-10-10T00:58:33.5328504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5328594Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5328864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5328946Z return func(*args, **kwargs) 2025-10-10T00:58:33.5329198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5329293Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5329547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5329620Z return func(*args, **kwargs) 2025-10-10T00:58:33.5329884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5329976Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5330237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5330312Z return func(*args, **kwargs) 2025-10-10T00:58:33.5330564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5330657Z value_states = self.v(current_states) 2025-10-10T00:58:33.5330661Z 2025-10-10T00:58:33.5330748Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5330842Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5330954Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5331171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5331241Z res = mod(**inputs) 2025-10-10T00:58:33.5331700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5331798Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5332058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5332200Z layer_outputs = layer_module( 2025-10-10T00:58:33.5332439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5332528Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5332817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5332893Z return func(*args, **kwargs) 2025-10-10T00:58:33.5333155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5333242Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5333549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5333636Z return func(*args, **kwargs) 2025-10-10T00:58:33.5333894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5333994Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5334249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5334332Z return func(*args, **kwargs) 2025-10-10T00:58:33.5334586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5334670Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5334674Z 2025-10-10T00:58:33.5334793Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5335007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5335083Z res = mod(**inputs) 2025-10-10T00:58:33.5335341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5335421Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5335685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5335757Z layer_outputs = layer_module( 2025-10-10T00:58:33.5335993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5336076Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5336315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5336391Z return func(*args, **kwargs) 2025-10-10T00:58:33.5336631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5336723Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5336967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5337044Z return func(*args, **kwargs) 2025-10-10T00:58:33.5337284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5337374Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5337622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5337694Z return func(*args, **kwargs) 2025-10-10T00:58:33.5337944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5338027Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5338030Z 2025-10-10T00:58:33.5338136Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5338345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5338430Z res = mod(**inputs) 2025-10-10T00:58:33.5338684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5338758Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5339019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5339103Z layer_outputs = layer_module( 2025-10-10T00:58:33.5339328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5339417Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5339691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5339773Z return func(*args, **kwargs) 2025-10-10T00:58:33.5340016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5340102Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5340356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5340426Z return func(*args, **kwargs) 2025-10-10T00:58:33.5340675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5340766Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5341025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5341113Z return func(*args, **kwargs) 2025-10-10T00:58:33.5341374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5341467Z key_states = self.k(current_states) 2025-10-10T00:58:33.5341470Z 2025-10-10T00:58:33.5341583Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5341795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5341872Z res = mod(**inputs) 2025-10-10T00:58:33.5342139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5342228Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5342499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5342586Z layer_outputs = layer_module( 2025-10-10T00:58:33.5342838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5342925Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5343202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5343281Z return func(*args, **kwargs) 2025-10-10T00:58:33.5343556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5343648Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5343916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5343999Z return func(*args, **kwargs) 2025-10-10T00:58:33.5344266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5344372Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5344646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5344741Z return func(*args, **kwargs) 2025-10-10T00:58:33.5345004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5345089Z value_states = self.v(current_states) 2025-10-10T00:58:33.5345093Z 2025-10-10T00:58:33.5345188Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5345293Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5345418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5345635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5345706Z res = mod(**inputs) 2025-10-10T00:58:33.5346003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5346103Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5346605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5346705Z layer_outputs = layer_module( 2025-10-10T00:58:33.5346949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5347048Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5347316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5347402Z return func(*args, **kwargs) 2025-10-10T00:58:33.5347667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5347766Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5348015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5348087Z return func(*args, **kwargs) 2025-10-10T00:58:33.5348336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5348425Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5348676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5348748Z return func(*args, **kwargs) 2025-10-10T00:58:33.5348989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5349079Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5349082Z 2025-10-10T00:58:33.5349191Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5349411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5349483Z res = mod(**inputs) 2025-10-10T00:58:33.5349741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5349841Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5350085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5350164Z layer_outputs = layer_module( 2025-10-10T00:58:33.5350386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5350466Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5350716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5350786Z return func(*args, **kwargs) 2025-10-10T00:58:33.5351034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5351119Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5351368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5351474Z return func(*args, **kwargs) 2025-10-10T00:58:33.5351720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 528, in forward 2025-10-10T00:58:33.5351889Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5351894Z 2025-10-10T00:58:33.5352005Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5352222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5352293Z res = mod(**inputs) 2025-10-10T00:58:33.5352573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5352676Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5352933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5353030Z layer_outputs = layer_module( 2025-10-10T00:58:33.5353253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5353334Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5353582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5353657Z return func(*args, **kwargs) 2025-10-10T00:58:33.5353919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5354020Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5354285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5354412Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5354666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5354781Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5354785Z 2025-10-10T00:58:33.5354895Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5355117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5355188Z res = mod(**inputs) 2025-10-10T00:58:33.5355447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5355534Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5355795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5355885Z layer_outputs = layer_module( 2025-10-10T00:58:33.5356121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5356216Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5356469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5356544Z return func(*args, **kwargs) 2025-10-10T00:58:33.5356805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5356905Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5357164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5357290Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5357545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5357671Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5357675Z 2025-10-10T00:58:33.5357787Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5358003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5358074Z res = mod(**inputs) 2025-10-10T00:58:33.5358358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5358438Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5358695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5358781Z layer_outputs = layer_module( 2025-10-10T00:58:33.5359053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5359148Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5359404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5359479Z return func(*args, **kwargs) 2025-10-10T00:58:33.5359742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5359839Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5360104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5360226Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5360482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5360587Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5360591Z 2025-10-10T00:58:33.5360702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5360924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5360994Z res = mod(**inputs) 2025-10-10T00:58:33.5361261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5361340Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5361600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5361687Z layer_outputs = layer_module( 2025-10-10T00:58:33.5361924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5362016Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5362278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5362355Z return func(*args, **kwargs) 2025-10-10T00:58:33.5362618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5362714Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5362975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5363097Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5363357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5363446Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5363450Z 2025-10-10T00:58:33.5363560Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5363781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5363852Z res = mod(**inputs) 2025-10-10T00:58:33.5364138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5364217Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5364475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5364578Z layer_outputs = layer_module( 2025-10-10T00:58:33.5364813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5364905Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5365160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5365272Z return func(*args, **kwargs) 2025-10-10T00:58:33.5365536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5365626Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5365888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5365964Z return func(*args, **kwargs) 2025-10-10T00:58:33.5366226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5366316Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5366571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5366652Z return func(*args, **kwargs) 2025-10-10T00:58:33.5366908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5367000Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5367004Z 2025-10-10T00:58:33.5367113Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5367324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5367403Z res = mod(**inputs) 2025-10-10T00:58:33.5367659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5367747Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5368006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5368084Z layer_outputs = layer_module( 2025-10-10T00:58:33.5368325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5368415Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5368677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5368754Z return func(*args, **kwargs) 2025-10-10T00:58:33.5369016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5369106Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5369361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5369444Z return func(*args, **kwargs) 2025-10-10T00:58:33.5369696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5369795Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5370051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5370128Z return func(*args, **kwargs) 2025-10-10T00:58:33.5370390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5370494Z key_states = self.k(current_states) 2025-10-10T00:58:33.5370498Z 2025-10-10T00:58:33.5370615Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5370827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5370914Z res = mod(**inputs) 2025-10-10T00:58:33.5371182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5371259Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5371524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5371644Z layer_outputs = layer_module( 2025-10-10T00:58:33.5371886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5371973Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5372229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5372313Z return func(*args, **kwargs) 2025-10-10T00:58:33.5372567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5372662Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5372920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5372994Z return func(*args, **kwargs) 2025-10-10T00:58:33.5373258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5373348Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5373609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5373684Z return func(*args, **kwargs) 2025-10-10T00:58:33.5373937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5374046Z value_states = self.v(current_states) 2025-10-10T00:58:33.5374050Z 2025-10-10T00:58:33.5374141Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5374235Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5374344Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5374566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5374636Z res = mod(**inputs) 2025-10-10T00:58:33.5374903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5374992Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5375258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5375357Z layer_outputs = layer_module( 2025-10-10T00:58:33.5375591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5375676Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5375938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5376013Z return func(*args, **kwargs) 2025-10-10T00:58:33.5376273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5376360Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5376619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5376700Z return func(*args, **kwargs) 2025-10-10T00:58:33.5376979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5377076Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5377331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5377429Z return func(*args, **kwargs) 2025-10-10T00:58:33.5377685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5377770Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5377774Z 2025-10-10T00:58:33.5377893Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5378145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5378224Z res = mod(**inputs) 2025-10-10T00:58:33.5378482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5378561Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5378830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5378903Z layer_outputs = layer_module( 2025-10-10T00:58:33.5379134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5379217Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5379466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5379547Z return func(*args, **kwargs) 2025-10-10T00:58:33.5379804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5379901Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5380159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5380240Z return func(*args, **kwargs) 2025-10-10T00:58:33.5380497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5380592Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5380852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5380925Z return func(*args, **kwargs) 2025-10-10T00:58:33.5381186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5381274Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5381278Z 2025-10-10T00:58:33.5381389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5381621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5381691Z res = mod(**inputs) 2025-10-10T00:58:33.5381972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5382051Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5382329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5382415Z layer_outputs = layer_module( 2025-10-10T00:58:33.5382665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5382762Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5383041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5383125Z return func(*args, **kwargs) 2025-10-10T00:58:33.5383418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5383511Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5383798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5383890Z return func(*args, **kwargs) 2025-10-10T00:58:33.5384168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5384264Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5384538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5384658Z return func(*args, **kwargs) 2025-10-10T00:58:33.5384927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5385023Z key_states = self.k(current_states) 2025-10-10T00:58:33.5385027Z 2025-10-10T00:58:33.5385142Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5385377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5385449Z res = mod(**inputs) 2025-10-10T00:58:33.5385741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5385832Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5386113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5386199Z layer_outputs = layer_module( 2025-10-10T00:58:33.5386534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5386628Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5386912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5386991Z return func(*args, **kwargs) 2025-10-10T00:58:33.5387271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5387363Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5387639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5387727Z return func(*args, **kwargs) 2025-10-10T00:58:33.5387998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5388104Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5388377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5388465Z return func(*args, **kwargs) 2025-10-10T00:58:33.5388739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5388829Z value_states = self.v(current_states) 2025-10-10T00:58:33.5388833Z 2025-10-10T00:58:33.5388936Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5389027Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5389151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5389378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5389452Z res = mod(**inputs) 2025-10-10T00:58:33.5389749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5389835Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5390109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5390215Z layer_outputs = layer_module( 2025-10-10T00:58:33.5390460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5390556Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5390848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5390935Z return func(*args, **kwargs) 2025-10-10T00:58:33.5391207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5391297Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5391615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5391695Z return func(*args, **kwargs) 2025-10-10T00:58:33.5391975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5392068Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5392356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5392433Z return func(*args, **kwargs) 2025-10-10T00:58:33.5392731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5392824Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5392828Z 2025-10-10T00:58:33.5392944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5393186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5393258Z res = mod(**inputs) 2025-10-10T00:58:33.5393530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5393621Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5393894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5393981Z layer_outputs = layer_module( 2025-10-10T00:58:33.5394231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5394317Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5394635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5394709Z return func(*args, **kwargs) 2025-10-10T00:58:33.5394992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5395094Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5395380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5395510Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5395787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5395908Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5395912Z 2025-10-10T00:58:33.5396025Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5396258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5396329Z res = mod(**inputs) 2025-10-10T00:58:33.5396623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5396711Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5396996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5397084Z layer_outputs = layer_module( 2025-10-10T00:58:33.5397337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5397450Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5397720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5397806Z return func(*args, **kwargs) 2025-10-10T00:58:33.5398075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5398222Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5398497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5398623Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5398887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5398981Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5398985Z 2025-10-10T00:58:33.5399097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5399315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5399385Z res = mod(**inputs) 2025-10-10T00:58:33.5399645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5399735Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5400005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5400092Z layer_outputs = layer_module( 2025-10-10T00:58:33.5400341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5400437Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5400714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5400793Z return func(*args, **kwargs) 2025-10-10T00:58:33.5401065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5401167Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5401444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5401574Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5401849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5401955Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5401959Z 2025-10-10T00:58:33.5402073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5402302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5402376Z res = mod(**inputs) 2025-10-10T00:58:33.5402650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5402731Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5402999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5403090Z layer_outputs = layer_module( 2025-10-10T00:58:33.5403336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5403430Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5403723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5403798Z return func(*args, **kwargs) 2025-10-10T00:58:33.5404087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5404186Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5404452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5404573Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5404854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5404972Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5404976Z 2025-10-10T00:58:33.5405089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5405318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5405390Z res = mod(**inputs) 2025-10-10T00:58:33.5405666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5405749Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5406015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5406101Z layer_outputs = layer_module( 2025-10-10T00:58:33.5406348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5406448Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5406732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5406810Z return func(*args, **kwargs) 2025-10-10T00:58:33.5407098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5407195Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5407467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 218, in forward 2025-10-10T00:58:33.5407608Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-10-10T00:58:33.5407612Z 2025-10-10T00:58:33.5407732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5407943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5408016Z res = mod(**inputs) 2025-10-10T00:58:33.5408290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5408372Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5408650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5408730Z layer_outputs = layer_module( 2025-10-10T00:58:33.5408978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5409077Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5409344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5409429Z return func(*args, **kwargs) 2025-10-10T00:58:33.5409697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5409789Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5410063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5410160Z return func(*args, **kwargs) 2025-10-10T00:58:33.5410430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5410524Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5410810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5410889Z return func(*args, **kwargs) 2025-10-10T00:58:33.5411152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5411245Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5411249Z 2025-10-10T00:58:33.5411400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5411625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5411699Z res = mod(**inputs) 2025-10-10T00:58:33.5411968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5412056Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5412325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5412412Z layer_outputs = layer_module( 2025-10-10T00:58:33.5412658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5412747Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5413022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5413098Z return func(*args, **kwargs) 2025-10-10T00:58:33.5413370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5413462Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5413735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5413810Z return func(*args, **kwargs) 2025-10-10T00:58:33.5414087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5414187Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5414445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5414523Z return func(*args, **kwargs) 2025-10-10T00:58:33.5414788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5414876Z key_states = self.k(current_states) 2025-10-10T00:58:33.5414880Z 2025-10-10T00:58:33.5415001Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5415218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5415298Z res = mod(**inputs) 2025-10-10T00:58:33.5415568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5415656Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5415926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5416007Z layer_outputs = layer_module( 2025-10-10T00:58:33.5416263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5416357Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5416631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5416735Z return func(*args, **kwargs) 2025-10-10T00:58:33.5416996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5417094Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5417377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5417462Z return func(*args, **kwargs) 2025-10-10T00:58:33.5417728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5417820Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5418111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5418207Z return func(*args, **kwargs) 2025-10-10T00:58:33.5418476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5418565Z value_states = self.v(current_states) 2025-10-10T00:58:33.5418569Z 2025-10-10T00:58:33.5418667Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5418758Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5418874Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5419101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5419172Z res = mod(**inputs) 2025-10-10T00:58:33.5419449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5419530Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5419802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5419903Z layer_outputs = layer_module( 2025-10-10T00:58:33.5420150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5420245Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5420511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5420589Z return func(*args, **kwargs) 2025-10-10T00:58:33.5420861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5420952Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5421222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5421302Z return func(*args, **kwargs) 2025-10-10T00:58:33.5421563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5421664Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5421925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5422010Z return func(*args, **kwargs) 2025-10-10T00:58:33.5422276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5422370Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5422373Z 2025-10-10T00:58:33.5422488Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5422705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5422785Z res = mod(**inputs) 2025-10-10T00:58:33.5423054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5423143Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5423454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5423533Z layer_outputs = layer_module( 2025-10-10T00:58:33.5423784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5423900Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5424175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5424253Z return func(*args, **kwargs) 2025-10-10T00:58:33.5424516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5424653Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5424919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5425004Z return func(*args, **kwargs) 2025-10-10T00:58:33.5425268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5425370Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5425635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5425712Z return func(*args, **kwargs) 2025-10-10T00:58:33.5425982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5426069Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5426073Z 2025-10-10T00:58:33.5426195Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5426490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5426570Z res = mod(**inputs) 2025-10-10T00:58:33.5426857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5426938Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5427213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5427295Z layer_outputs = layer_module( 2025-10-10T00:58:33.5427551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5427641Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5427912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5428002Z return func(*args, **kwargs) 2025-10-10T00:58:33.5428267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5428368Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5428636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5428712Z return func(*args, **kwargs) 2025-10-10T00:58:33.5428987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5429082Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5429358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5429435Z return func(*args, **kwargs) 2025-10-10T00:58:33.5429704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5429800Z key_states = self.k(current_states) 2025-10-10T00:58:33.5429804Z 2025-10-10T00:58:33.5429920Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5430172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5430243Z res = mod(**inputs) 2025-10-10T00:58:33.5430519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5430619Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5430883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5430970Z layer_outputs = layer_module( 2025-10-10T00:58:33.5431213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5431342Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5431732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5431816Z return func(*args, **kwargs) 2025-10-10T00:58:33.5432080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5432167Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5432432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5432507Z return func(*args, **kwargs) 2025-10-10T00:58:33.5432760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5432863Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5433122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5433205Z return func(*args, **kwargs) 2025-10-10T00:58:33.5433461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5433559Z value_states = self.v(current_states) 2025-10-10T00:58:33.5433563Z 2025-10-10T00:58:33.5433652Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5433739Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5433860Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5434073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5434153Z res = mod(**inputs) 2025-10-10T00:58:33.5434412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5434492Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5434762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5434841Z layer_outputs = layer_module( 2025-10-10T00:58:33.5435088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5435172Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5435425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5435511Z return func(*args, **kwargs) 2025-10-10T00:58:33.5435767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5435862Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5436116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5436201Z return func(*args, **kwargs) 2025-10-10T00:58:33.5436454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5436596Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5436861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5436935Z return func(*args, **kwargs) 2025-10-10T00:58:33.5437225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5437313Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5437316Z 2025-10-10T00:58:33.5437426Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5437643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5437713Z res = mod(**inputs) 2025-10-10T00:58:33.5438039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5438121Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5438384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5438465Z layer_outputs = layer_module( 2025-10-10T00:58:33.5438690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5438779Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5439022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5439099Z return func(*args, **kwargs) 2025-10-10T00:58:33.5439345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5439443Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5439694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5439816Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5440068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5440174Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5440178Z 2025-10-10T00:58:33.5440292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5440520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5440593Z res = mod(**inputs) 2025-10-10T00:58:33.5440878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5440957Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5441222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5441309Z layer_outputs = layer_module( 2025-10-10T00:58:33.5441549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5441642Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5441911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5441993Z return func(*args, **kwargs) 2025-10-10T00:58:33.5442248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5442352Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5442603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5442722Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5442970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5443072Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5443076Z 2025-10-10T00:58:33.5443181Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5443388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5443470Z res = mod(**inputs) 2025-10-10T00:58:33.5443722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5443795Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5444099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5444214Z layer_outputs = layer_module( 2025-10-10T00:58:33.5444453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5444548Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5444813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5444894Z return func(*args, **kwargs) 2025-10-10T00:58:33.5445159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5445256Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5445529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5445653Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5445921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5446018Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5446022Z 2025-10-10T00:58:33.5446146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5446346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5446414Z res = mod(**inputs) 2025-10-10T00:58:33.5446666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5446742Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5446994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5447069Z layer_outputs = layer_module( 2025-10-10T00:58:33.5447293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5447387Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5447628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5447708Z return func(*args, **kwargs) 2025-10-10T00:58:33.5447952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5448045Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5448296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5448413Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5448664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5448747Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5448752Z 2025-10-10T00:58:33.5448866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5449065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5449158Z res = mod(**inputs) 2025-10-10T00:58:33.5449409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5449486Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5449754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5449830Z layer_outputs = layer_module( 2025-10-10T00:58:33.5450054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5450144Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5450407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5450502Z return func(*args, **kwargs) 2025-10-10T00:58:33.5450745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5450830Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5451082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5451152Z return func(*args, **kwargs) 2025-10-10T00:58:33.5451402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5451490Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5451755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5451830Z return func(*args, **kwargs) 2025-10-10T00:58:33.5452092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5452185Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5452190Z 2025-10-10T00:58:33.5452303Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5452528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5452601Z res = mod(**inputs) 2025-10-10T00:58:33.5452871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5452961Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5453228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5453316Z layer_outputs = layer_module( 2025-10-10T00:58:33.5453566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5453657Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5453929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5454007Z return func(*args, **kwargs) 2025-10-10T00:58:33.5454280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5454380Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5454648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5454723Z return func(*args, **kwargs) 2025-10-10T00:58:33.5454978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5455077Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5455337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5455418Z return func(*args, **kwargs) 2025-10-10T00:58:33.5455700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5455786Z key_states = self.k(current_states) 2025-10-10T00:58:33.5455790Z 2025-10-10T00:58:33.5455910Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5456145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5456227Z res = mod(**inputs) 2025-10-10T00:58:33.5456498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5456579Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5456873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5456969Z layer_outputs = layer_module( 2025-10-10T00:58:33.5457219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5457310Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5457582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5457658Z return func(*args, **kwargs) 2025-10-10T00:58:33.5457924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5458020Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5458281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5458362Z return func(*args, **kwargs) 2025-10-10T00:58:33.5458628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5458719Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5458993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5459069Z return func(*args, **kwargs) 2025-10-10T00:58:33.5459340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5459427Z value_states = self.v(current_states) 2025-10-10T00:58:33.5459431Z 2025-10-10T00:58:33.5459526Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5459614Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5459727Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5459950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5460026Z res = mod(**inputs) 2025-10-10T00:58:33.5460300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5460381Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5460648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5460734Z layer_outputs = layer_module( 2025-10-10T00:58:33.5460978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5461075Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5461340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5461416Z return func(*args, **kwargs) 2025-10-10T00:58:33.5461688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5461779Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5462046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5462144Z return func(*args, **kwargs) 2025-10-10T00:58:33.5462405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5462506Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5462793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5462877Z return func(*args, **kwargs) 2025-10-10T00:58:33.5463145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5463240Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5463243Z 2025-10-10T00:58:33.5463395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5463617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5463698Z res = mod(**inputs) 2025-10-10T00:58:33.5463969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5464058Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5464335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5464414Z layer_outputs = layer_module( 2025-10-10T00:58:33.5464660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5464747Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5465012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5465089Z return func(*args, **kwargs) 2025-10-10T00:58:33.5465347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5465443Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5465702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5465784Z return func(*args, **kwargs) 2025-10-10T00:58:33.5466044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-10-10T00:58:33.5466198Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5466202Z 2025-10-10T00:58:33.5466391Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5466621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5466710Z res = mod(**inputs) 2025-10-10T00:58:33.5466978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5467071Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5467337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5467418Z layer_outputs = layer_module( 2025-10-10T00:58:33.5467674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5467763Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5468042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5468112Z return func(*args, **kwargs) 2025-10-10T00:58:33.5468362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5468449Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5468692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5468794Z return func(*args, **kwargs) 2025-10-10T00:58:33.5469039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5469137Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5469404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5469477Z return func(*args, **kwargs) 2025-10-10T00:58:33.5469728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5469811Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5469831Z 2025-10-10T00:58:33.5469969Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5470172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5470241Z res = mod(**inputs) 2025-10-10T00:58:33.5470495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5470570Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5470832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5470909Z layer_outputs = layer_module( 2025-10-10T00:58:33.5471158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5471244Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5471510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5471592Z return func(*args, **kwargs) 2025-10-10T00:58:33.5471837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5471933Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5472196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5472273Z return func(*args, **kwargs) 2025-10-10T00:58:33.5472545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5472647Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5472901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5472975Z return func(*args, **kwargs) 2025-10-10T00:58:33.5473224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5473313Z key_states = self.k(current_states) 2025-10-10T00:58:33.5473319Z 2025-10-10T00:58:33.5473426Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5473638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5473709Z res = mod(**inputs) 2025-10-10T00:58:33.5473977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5474060Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5474325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5474413Z layer_outputs = layer_module( 2025-10-10T00:58:33.5474658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5474755Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5475016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5475108Z return func(*args, **kwargs) 2025-10-10T00:58:33.5475380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5475468Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5475754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5475831Z return func(*args, **kwargs) 2025-10-10T00:58:33.5476086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5476184Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5476476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5476558Z return func(*args, **kwargs) 2025-10-10T00:58:33.5476817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5476910Z value_states = self.v(current_states) 2025-10-10T00:58:33.5476914Z 2025-10-10T00:58:33.5477000Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5477087Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5477210Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5477425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5477504Z res = mod(**inputs) 2025-10-10T00:58:33.5477766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5477849Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5478118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5478199Z layer_outputs = layer_module( 2025-10-10T00:58:33.5478448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5478533Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5478795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5478877Z return func(*args, **kwargs) 2025-10-10T00:58:33.5479136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5479232Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5479494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5479577Z return func(*args, **kwargs) 2025-10-10T00:58:33.5479836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5479930Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5480207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5480277Z return func(*args, **kwargs) 2025-10-10T00:58:33.5480530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5480609Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5480613Z 2025-10-10T00:58:33.5480719Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5480928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5480997Z res = mod(**inputs) 2025-10-10T00:58:33.5481262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5481340Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5482460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5482546Z layer_outputs = layer_module( 2025-10-10T00:58:33.5482789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5482904Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5483164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5483249Z return func(*args, **kwargs) 2025-10-10T00:58:33.5483507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5483644Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5483907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5484035Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5484298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5484407Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5484411Z 2025-10-10T00:58:33.5484524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5484743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5484816Z res = mod(**inputs) 2025-10-10T00:58:33.5485082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5485166Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5485432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5485513Z layer_outputs = layer_module( 2025-10-10T00:58:33.5485752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5485846Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5486111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5486194Z return func(*args, **kwargs) 2025-10-10T00:58:33.5486460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5486558Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5486833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5486958Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5487227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5487316Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5487320Z 2025-10-10T00:58:33.5487430Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5487670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5487739Z res = mod(**inputs) 2025-10-10T00:58:33.5488002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5488081Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5488383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5488466Z layer_outputs = layer_module( 2025-10-10T00:58:33.5488706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5488823Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5489100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5489182Z return func(*args, **kwargs) 2025-10-10T00:58:33.5489466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5489577Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5489848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5489969Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5490277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5490373Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5490379Z 2025-10-10T00:58:33.5490497Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5490727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5490798Z res = mod(**inputs) 2025-10-10T00:58:33.5491071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5491149Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5491412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5491490Z layer_outputs = layer_module( 2025-10-10T00:58:33.5491731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5491826Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5492084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5492167Z return func(*args, **kwargs) 2025-10-10T00:58:33.5492422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5492519Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5492784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5492908Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5493169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5493259Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5493264Z 2025-10-10T00:58:33.5493382Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5493596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5493669Z res = mod(**inputs) 2025-10-10T00:58:33.5493938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5494018Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5494284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5494366Z layer_outputs = layer_module( 2025-10-10T00:58:33.5494605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5494700Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5494960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5495045Z return func(*args, **kwargs) 2025-10-10T00:58:33.5495301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5495408Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5495670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5495744Z return func(*args, **kwargs) 2025-10-10T00:58:33.5496023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5496117Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5496381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5496457Z return func(*args, **kwargs) 2025-10-10T00:58:33.5496762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5496861Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5496866Z 2025-10-10T00:58:33.5496980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5497205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5497277Z res = mod(**inputs) 2025-10-10T00:58:33.5497544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5497634Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5497902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5497986Z layer_outputs = layer_module( 2025-10-10T00:58:33.5498222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5498310Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5498573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5498648Z return func(*args, **kwargs) 2025-10-10T00:58:33.5498908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5498996Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5499259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5499333Z return func(*args, **kwargs) 2025-10-10T00:58:33.5499587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5499684Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5499946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5500026Z return func(*args, **kwargs) 2025-10-10T00:58:33.5500280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5500362Z key_states = self.k(current_states) 2025-10-10T00:58:33.5500366Z 2025-10-10T00:58:33.5500483Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5500695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5500771Z res = mod(**inputs) 2025-10-10T00:58:33.5501029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5501112Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5501374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5501454Z layer_outputs = layer_module( 2025-10-10T00:58:33.5501697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5501801Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5502068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5502142Z return func(*args, **kwargs) 2025-10-10T00:58:33.5502417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5502515Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5502769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5502851Z return func(*args, **kwargs) 2025-10-10T00:58:33.5503143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5503235Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5503502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5503576Z return func(*args, **kwargs) 2025-10-10T00:58:33.5503840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5503926Z value_states = self.v(current_states) 2025-10-10T00:58:33.5503930Z 2025-10-10T00:58:33.5504025Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5504112Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5504222Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5504448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5504522Z res = mod(**inputs) 2025-10-10T00:58:33.5504799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5504881Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5505151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5505236Z layer_outputs = layer_module( 2025-10-10T00:58:33.5505482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5505577Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5505841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5505915Z return func(*args, **kwargs) 2025-10-10T00:58:33.5506190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-10-10T00:58:33.5506281Z self_attention_outputs = self.layer[0]( 2025-10-10T00:58:33.5506644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5506730Z return func(*args, **kwargs) 2025-10-10T00:58:33.5506995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-10-10T00:58:33.5507095Z attention_output = self.SelfAttention( 2025-10-10T00:58:33.5507363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5507448Z return func(*args, **kwargs) 2025-10-10T00:58:33.5507709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5507805Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5507813Z 2025-10-10T00:58:33.5507929Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5508147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5508255Z res = mod(**inputs) 2025-10-10T00:58:33.5508522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5508615Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5508903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5508986Z layer_outputs = layer_module( 2025-10-10T00:58:33.5509239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5509328Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5509614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5509710Z return func(*args, **kwargs) 2025-10-10T00:58:33.5509979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5510074Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5510339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5510425Z return func(*args, **kwargs) 2025-10-10T00:58:33.5510691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5510794Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5511061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5511138Z return func(*args, **kwargs) 2025-10-10T00:58:33.5511414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-10-10T00:58:33.5511501Z query_states = self.q(hidden_states) 2025-10-10T00:58:33.5511507Z 2025-10-10T00:58:33.5511633Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5511850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5511922Z res = mod(**inputs) 2025-10-10T00:58:33.5512199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5512281Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5512557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5512637Z layer_outputs = layer_module( 2025-10-10T00:58:33.5512889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5512979Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5513251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5513336Z return func(*args, **kwargs) 2025-10-10T00:58:33.5513598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5513697Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5513963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5514040Z return func(*args, **kwargs) 2025-10-10T00:58:33.5514312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5514406Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5514681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5514756Z return func(*args, **kwargs) 2025-10-10T00:58:33.5515035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-10-10T00:58:33.5515128Z key_states = self.k(current_states) 2025-10-10T00:58:33.5515133Z 2025-10-10T00:58:33.5515245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5515487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5515562Z res = mod(**inputs) 2025-10-10T00:58:33.5515839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5515922Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5516204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5516310Z layer_outputs = layer_module( 2025-10-10T00:58:33.5516552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5516646Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5516912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5516989Z return func(*args, **kwargs) 2025-10-10T00:58:33.5517264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5517355Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5517625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5517700Z return func(*args, **kwargs) 2025-10-10T00:58:33.5517965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5518067Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5518333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5518416Z return func(*args, **kwargs) 2025-10-10T00:58:33.5518678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-10-10T00:58:33.5518775Z value_states = self.v(current_states) 2025-10-10T00:58:33.5518780Z 2025-10-10T00:58:33.5518869Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5518957Z cudagraph partition due to non gpu ops 2025-10-10T00:58:33.5519080Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5519300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5519383Z res = mod(**inputs) 2025-10-10T00:58:33.5519650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5519734Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5520007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5520087Z layer_outputs = layer_module( 2025-10-10T00:58:33.5520339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5520427Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5520687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5520771Z return func(*args, **kwargs) 2025-10-10T00:58:33.5521035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5521133Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5521400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5521501Z return func(*args, **kwargs) 2025-10-10T00:58:33.5521770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-10-10T00:58:33.5521862Z attention_output = self.EncDecAttention( 2025-10-10T00:58:33.5522162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5522240Z return func(*args, **kwargs) 2025-10-10T00:58:33.5522517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-10-10T00:58:33.5522602Z attn_output = self.o(attn_output) 2025-10-10T00:58:33.5522623Z 2025-10-10T00:58:33.5522775Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5522995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5523068Z res = mod(**inputs) 2025-10-10T00:58:33.5523333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5523412Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5523669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5523754Z layer_outputs = layer_module( 2025-10-10T00:58:33.5523992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5524083Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5524386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5524468Z return func(*args, **kwargs) 2025-10-10T00:58:33.5524725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-10-10T00:58:33.5524817Z cross_attention_outputs = self.layer[1]( 2025-10-10T00:58:33.5525078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5525152Z return func(*args, **kwargs) 2025-10-10T00:58:33.5525415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 528, in forward 2025-10-10T00:58:33.5525556Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-10-10T00:58:33.5525560Z 2025-10-10T00:58:33.5525671Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5525890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5525964Z res = mod(**inputs) 2025-10-10T00:58:33.5526230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5526310Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5526567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5526654Z layer_outputs = layer_module( 2025-10-10T00:58:33.5526891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5526985Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5527241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5527323Z return func(*args, **kwargs) 2025-10-10T00:58:33.5527582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5527684Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5527947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5528091Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5528352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-10-10T00:58:33.5528478Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-10-10T00:58:33.5528482Z 2025-10-10T00:58:33.5528593Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5528812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5528882Z res = mod(**inputs) 2025-10-10T00:58:33.5529166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5529265Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5529529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5529610Z layer_outputs = layer_module( 2025-10-10T00:58:33.5529845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5529938Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5530206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5530287Z return func(*args, **kwargs) 2025-10-10T00:58:33.5530552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5530650Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5530937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5531061Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5531323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-10-10T00:58:33.5531410Z hidden_linear = self.wi_1(hidden_states) 2025-10-10T00:58:33.5531414Z 2025-10-10T00:58:33.5531657Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5531916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5531990Z res = mod(**inputs) 2025-10-10T00:58:33.5532259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5532338Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5532606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5532686Z layer_outputs = layer_module( 2025-10-10T00:58:33.5532924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5533020Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5533278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5533362Z return func(*args, **kwargs) 2025-10-10T00:58:33.5533619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5533717Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5533981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5534107Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5534372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-10-10T00:58:33.5534519Z hidden_states = hidden_gelu * hidden_linear 2025-10-10T00:58:33.5534523Z 2025-10-10T00:58:33.5534642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5534854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5534925Z res = mod(**inputs) 2025-10-10T00:58:33.5535216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-10-10T00:58:33.5535297Z decoder_outputs = self.decoder( 2025-10-10T00:58:33.5535562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-10-10T00:58:33.5535639Z layer_outputs = layer_module( 2025-10-10T00:58:33.5535926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:58:33.5536024Z return super().__call__(*args, **kwargs) 2025-10-10T00:58:33.5536286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:58:33.5536367Z return func(*args, **kwargs) 2025-10-10T00:58:33.5536623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-10-10T00:58:33.5536721Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T00:58:33.5536980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-10-10T00:58:33.5537104Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T00:58:33.5537363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-10-10T00:58:33.5537454Z hidden_states = self.wo(hidden_states) 2025-10-10T00:58:33.5537457Z 2025-10-10T00:58:33.5537575Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5537787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5537858Z res = mod(**inputs) 2025-10-10T00:58:33.5538124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1815, in forward 2025-10-10T00:58:33.5538217Z lm_logits = self.lm_head(sequence_output) 2025-10-10T00:58:33.5538223Z 2025-10-10T00:58:33.5538340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:58:33.5538550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:58:33.5538619Z res = mod(**inputs) 2025-10-10T00:58:33.5538884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1822, in forward 2025-10-10T00:58:33.5539043Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-10-10T00:58:33.5539047Z 2025-10-10T00:58:44.4736156Z Compilation time (from dynamo_timed): 22.243234117 2025-10-10T00:58:44.4978702Z pass 2025-10-10T00:58:44.4979131Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:44.4979958Z TIMING: _recursive_pre_grad_passes:0.01525 _recursive_joint_graph_passes:0.87831 _recursive_post_grad_passes:0.10553 async_compile.wait:0.74166 code_gen:10.19554 inductor_compile:12.53102 backend_compile:17.76463 gc:0.00048 entire_frame_compile:22.24323 total_wall_time:22.24323 2025-10-10T00:58:44.4980997Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:16469 | FakeTensor.__torch_dispatch__:6387 | ProxyTorchDispatchMode.__torch_dispatch__:4859 2025-10-10T00:58:44.4981569Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-10-10T00:58:47.5680511Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:58:47.5682289Z import pynvml # type: ignore[import] 2025-10-10T00:58:51.1023180Z 2025-10-10T00:58:51.1163573Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-10-10T00:58:51.1164818Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-10-10T00:58:54.3511637Z 2025-10-10T00:58:54.3512636Z loading model: 0it [00:03, ?it/s] 2025-10-10T00:58:54.3545950Z cpu eval MegatronBertForCausalLM 2025-10-10T00:58:56.0187501Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:56.6422001Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:58:57.2718648Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:59:12.6797338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6797857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6798215Z res = mod(**inputs) 2025-10-10T00:59:12.6798703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6799153Z outputs = self.bert( 2025-10-10T00:59:12.6799571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6800019Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6800465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6800913Z layer_outputs = layer_module( 2025-10-10T00:59:12.6801292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6801667Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6802070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6802461Z return func(*args, **kwargs) 2025-10-10T00:59:12.6803015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6803504Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6803937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6804351Z return func(*args, **kwargs) 2025-10-10T00:59:12.6804805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6805262Z self_outputs = self.self( 2025-10-10T00:59:12.6805662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6806063Z return func(*args, **kwargs) 2025-10-10T00:59:12.6806522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.6806992Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.6807150Z 2025-10-10T00:59:12.6807277Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6807678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6808051Z res = mod(**inputs) 2025-10-10T00:59:12.6808494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6809298Z outputs = self.bert( 2025-10-10T00:59:12.6809726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6810175Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6810696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6811152Z layer_outputs = layer_module( 2025-10-10T00:59:12.6811535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6811933Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6812389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6812847Z return func(*args, **kwargs) 2025-10-10T00:59:12.6813290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6813759Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6814169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6814571Z return func(*args, **kwargs) 2025-10-10T00:59:12.6815054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6815506Z self_outputs = self.self( 2025-10-10T00:59:12.6815898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6816293Z return func(*args, **kwargs) 2025-10-10T00:59:12.6816741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.6817202Z key_layer = self.key(current_states) 2025-10-10T00:59:12.6817349Z 2025-10-10T00:59:12.6817475Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6817882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6818237Z res = mod(**inputs) 2025-10-10T00:59:12.6818671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6819121Z outputs = self.bert( 2025-10-10T00:59:12.6819549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6820002Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6820451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6820899Z layer_outputs = layer_module( 2025-10-10T00:59:12.6821279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6821674Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6822081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6822484Z return func(*args, **kwargs) 2025-10-10T00:59:12.6822931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6823406Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6823838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6824244Z return func(*args, **kwargs) 2025-10-10T00:59:12.6824702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6825192Z self_outputs = self.self( 2025-10-10T00:59:12.6825592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6826000Z return func(*args, **kwargs) 2025-10-10T00:59:12.6826639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.6827132Z value_layer = self.value(current_states) 2025-10-10T00:59:12.6827293Z 2025-10-10T00:59:12.6827387Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6827636Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6827887Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6828327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6828683Z res = mod(**inputs) 2025-10-10T00:59:12.6829124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6829575Z outputs = self.bert( 2025-10-10T00:59:12.6830003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6830464Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6830916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6831372Z layer_outputs = layer_module( 2025-10-10T00:59:12.6831930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6832343Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6832768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6833182Z return func(*args, **kwargs) 2025-10-10T00:59:12.6833633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6834100Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6834523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6834932Z return func(*args, **kwargs) 2025-10-10T00:59:12.6835387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.6835916Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.6836437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.6836919Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6837083Z 2025-10-10T00:59:12.6837200Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6837606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6837967Z res = mod(**inputs) 2025-10-10T00:59:12.6838402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6838872Z outputs = self.bert( 2025-10-10T00:59:12.6839311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6839781Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6840239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6840704Z layer_outputs = layer_module( 2025-10-10T00:59:12.6841127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6841523Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6841934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6842362Z return func(*args, **kwargs) 2025-10-10T00:59:12.6842810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6843279Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6843725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6844831Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6845274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.6845761Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.6846209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.6846642Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6846824Z 2025-10-10T00:59:12.6846931Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6847290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6847605Z res = mod(**inputs) 2025-10-10T00:59:12.6848002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6848428Z outputs = self.bert( 2025-10-10T00:59:12.6848839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6849305Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6849747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6850176Z layer_outputs = layer_module( 2025-10-10T00:59:12.6850535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6850907Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6851295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6851669Z return func(*args, **kwargs) 2025-10-10T00:59:12.6852093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6852544Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6852948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6853332Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6853778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.6854254Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.6854702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.6855161Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.6855548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.6855904Z return self.act(input) 2025-10-10T00:59:12.6856049Z 2025-10-10T00:59:12.6856166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6856534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6856856Z res = mod(**inputs) 2025-10-10T00:59:12.6857552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6857968Z outputs = self.bert( 2025-10-10T00:59:12.6858373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6858806Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6859280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6859747Z layer_outputs = layer_module( 2025-10-10T00:59:12.6860126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6860528Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6860949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6861370Z return func(*args, **kwargs) 2025-10-10T00:59:12.6861822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6862304Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6862758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6863202Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6863703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.6864268Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.6864794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.6865280Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6865433Z 2025-10-10T00:59:12.6865560Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6865952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6866312Z res = mod(**inputs) 2025-10-10T00:59:12.6866930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6867425Z outputs = self.bert( 2025-10-10T00:59:12.6867857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6868301Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6868731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6884782Z layer_outputs = layer_module( 2025-10-10T00:59:12.6885381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6885811Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6886260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6886692Z return func(*args, **kwargs) 2025-10-10T00:59:12.6887142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6887601Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6888135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6888549Z return func(*args, **kwargs) 2025-10-10T00:59:12.6889008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6889517Z self_outputs = self.self( 2025-10-10T00:59:12.6889915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6890323Z return func(*args, **kwargs) 2025-10-10T00:59:12.6890771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.6891318Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.6891478Z 2025-10-10T00:59:12.6891625Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6892028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6892388Z res = mod(**inputs) 2025-10-10T00:59:12.6892846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6893304Z outputs = self.bert( 2025-10-10T00:59:12.6893747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6894222Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6894697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6895176Z layer_outputs = layer_module( 2025-10-10T00:59:12.6895562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6895971Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6896405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6896822Z return func(*args, **kwargs) 2025-10-10T00:59:12.6897273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6897754Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6898180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6898588Z return func(*args, **kwargs) 2025-10-10T00:59:12.6899047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6899502Z self_outputs = self.self( 2025-10-10T00:59:12.6899895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6900311Z return func(*args, **kwargs) 2025-10-10T00:59:12.6900760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.6901235Z key_layer = self.key(current_states) 2025-10-10T00:59:12.6901385Z 2025-10-10T00:59:12.6901512Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6901904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6902263Z res = mod(**inputs) 2025-10-10T00:59:12.6902719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6903183Z outputs = self.bert( 2025-10-10T00:59:12.6903637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6904130Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6904618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6905079Z layer_outputs = layer_module( 2025-10-10T00:59:12.6905483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6905894Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6906414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6906866Z return func(*args, **kwargs) 2025-10-10T00:59:12.6907380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6907959Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6908376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6908790Z return func(*args, **kwargs) 2025-10-10T00:59:12.6909251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6909777Z self_outputs = self.self( 2025-10-10T00:59:12.6910187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6910608Z return func(*args, **kwargs) 2025-10-10T00:59:12.6911079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.6911548Z value_layer = self.value(current_states) 2025-10-10T00:59:12.6911698Z 2025-10-10T00:59:12.6911797Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6912040Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6912299Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6912698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6913060Z res = mod(**inputs) 2025-10-10T00:59:12.6913503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6913958Z outputs = self.bert( 2025-10-10T00:59:12.6914394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6914857Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6915340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6915818Z layer_outputs = layer_module( 2025-10-10T00:59:12.6916213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6916627Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6917059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6917488Z return func(*args, **kwargs) 2025-10-10T00:59:12.6917943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6918431Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6918868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6919297Z return func(*args, **kwargs) 2025-10-10T00:59:12.6919763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.6920310Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.6920847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.6921338Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6921511Z 2025-10-10T00:59:12.6921641Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6922047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6922401Z res = mod(**inputs) 2025-10-10T00:59:12.6922869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6923356Z outputs = self.bert( 2025-10-10T00:59:12.6923803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6924289Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6924754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6925226Z layer_outputs = layer_module( 2025-10-10T00:59:12.6925623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6926028Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6926447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6926872Z return func(*args, **kwargs) 2025-10-10T00:59:12.6927332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6927816Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6928275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6928714Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6929218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.6929762Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.6930267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.6930747Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6930904Z 2025-10-10T00:59:12.6931027Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6931435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6931978Z res = mod(**inputs) 2025-10-10T00:59:12.6932441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6932934Z outputs = self.bert( 2025-10-10T00:59:12.6933362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6933829Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6934305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6934836Z layer_outputs = layer_module( 2025-10-10T00:59:12.6935214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6935619Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6936039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6936543Z return func(*args, **kwargs) 2025-10-10T00:59:12.6936991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6937459Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6937958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6938395Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6938891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.6939468Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.6939954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.6940512Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.6940935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.6941316Z return self.act(input) 2025-10-10T00:59:12.6941440Z 2025-10-10T00:59:12.6941566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6941985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6942364Z res = mod(**inputs) 2025-10-10T00:59:12.6942826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6943303Z outputs = self.bert( 2025-10-10T00:59:12.6943744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6944223Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6944707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6945192Z layer_outputs = layer_module( 2025-10-10T00:59:12.6945591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6945992Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6946554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6946998Z return func(*args, **kwargs) 2025-10-10T00:59:12.6947468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6947954Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6948396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6948867Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6949369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.6949947Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.6950533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.6951016Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6951177Z 2025-10-10T00:59:12.6951297Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6951702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6952064Z res = mod(**inputs) 2025-10-10T00:59:12.6952524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6953033Z outputs = self.bert( 2025-10-10T00:59:12.6953444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6953894Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6954321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6954743Z layer_outputs = layer_module( 2025-10-10T00:59:12.6955109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6955522Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6955917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6956304Z return func(*args, **kwargs) 2025-10-10T00:59:12.6956734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.6957216Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.6957632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.6958030Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.6958487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.6959037Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.6959556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.6960019Z return input_tensor + hidden_states 2025-10-10T00:59:12.6960164Z 2025-10-10T00:59:12.6960279Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6960668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6960998Z res = mod(**inputs) 2025-10-10T00:59:12.6961432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6961889Z outputs = self.bert( 2025-10-10T00:59:12.6962314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6962775Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6963233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6963690Z layer_outputs = layer_module( 2025-10-10T00:59:12.6964068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6964452Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6964851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6965231Z return func(*args, **kwargs) 2025-10-10T00:59:12.6965647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6966083Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6966480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6966858Z return func(*args, **kwargs) 2025-10-10T00:59:12.6967280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6967731Z self_outputs = self.self( 2025-10-10T00:59:12.6968095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6968475Z return func(*args, **kwargs) 2025-10-10T00:59:12.6968909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.6969359Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.6969501Z 2025-10-10T00:59:12.6969618Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6970005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6970361Z res = mod(**inputs) 2025-10-10T00:59:12.6970756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6971173Z outputs = self.bert( 2025-10-10T00:59:12.6971557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6972060Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6972486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6972913Z layer_outputs = layer_module( 2025-10-10T00:59:12.6973270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6973640Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6974019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6974385Z return func(*args, **kwargs) 2025-10-10T00:59:12.6974791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6975214Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6975584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6975952Z return func(*args, **kwargs) 2025-10-10T00:59:12.6976367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6976788Z self_outputs = self.self( 2025-10-10T00:59:12.6977148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6977526Z return func(*args, **kwargs) 2025-10-10T00:59:12.6977937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.6978372Z key_layer = self.key(current_states) 2025-10-10T00:59:12.6978507Z 2025-10-10T00:59:12.6978623Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6978981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6979314Z res = mod(**inputs) 2025-10-10T00:59:12.6979736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6980186Z outputs = self.bert( 2025-10-10T00:59:12.6980613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6981063Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6981515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6982007Z layer_outputs = layer_module( 2025-10-10T00:59:12.6982388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6982788Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6983221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6983623Z return func(*args, **kwargs) 2025-10-10T00:59:12.6984067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6984527Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6984954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6985382Z return func(*args, **kwargs) 2025-10-10T00:59:12.6985869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.6986416Z self_outputs = self.self( 2025-10-10T00:59:12.6986837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6987250Z return func(*args, **kwargs) 2025-10-10T00:59:12.6987720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.6988197Z value_layer = self.value(current_states) 2025-10-10T00:59:12.6988345Z 2025-10-10T00:59:12.6988445Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6988674Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.6988939Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6989331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6989685Z res = mod(**inputs) 2025-10-10T00:59:12.6990118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6990570Z outputs = self.bert( 2025-10-10T00:59:12.6990998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.6991451Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.6991906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.6992365Z layer_outputs = layer_module( 2025-10-10T00:59:12.6992737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.6993133Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.6993540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6993939Z return func(*args, **kwargs) 2025-10-10T00:59:12.6994372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.6994849Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.6995265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.6995666Z return func(*args, **kwargs) 2025-10-10T00:59:12.6996112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.6996595Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.6997082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.6997560Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.6997701Z 2025-10-10T00:59:12.6997817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.6998186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.6998510Z res = mod(**inputs) 2025-10-10T00:59:12.6998935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.6999366Z outputs = self.bert( 2025-10-10T00:59:12.6999783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7000278Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7000705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7001139Z layer_outputs = layer_module( 2025-10-10T00:59:12.7001497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7001869Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7002248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7002629Z return func(*args, **kwargs) 2025-10-10T00:59:12.7003061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7003529Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7003968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7004387Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7004865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7005356Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7005813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7006253Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7006396Z 2025-10-10T00:59:12.7006506Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7006873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7007203Z res = mod(**inputs) 2025-10-10T00:59:12.7007606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7008056Z outputs = self.bert( 2025-10-10T00:59:12.7008478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7008932Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7009380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7009832Z layer_outputs = layer_module( 2025-10-10T00:59:12.7010202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7010601Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7011015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7011414Z return func(*args, **kwargs) 2025-10-10T00:59:12.7011855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7012355Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7012802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7013240Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7013754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7014278Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7014758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7015292Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7015711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7016087Z return self.act(input) 2025-10-10T00:59:12.7016211Z 2025-10-10T00:59:12.7016325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7016719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7017070Z res = mod(**inputs) 2025-10-10T00:59:12.7017503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7017956Z outputs = self.bert( 2025-10-10T00:59:12.7018376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7018828Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7019280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7019736Z layer_outputs = layer_module( 2025-10-10T00:59:12.7020116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7020500Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7020911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7021314Z return func(*args, **kwargs) 2025-10-10T00:59:12.7021753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7022219Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7022655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7023078Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7023562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7024109Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7024617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7025090Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7025252Z 2025-10-10T00:59:12.7025369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7025769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7026121Z res = mod(**inputs) 2025-10-10T00:59:12.7026647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7027126Z outputs = self.bert( 2025-10-10T00:59:12.7027572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7028077Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7028544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7029022Z layer_outputs = layer_module( 2025-10-10T00:59:12.7029434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7029830Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7030254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7030660Z return func(*args, **kwargs) 2025-10-10T00:59:12.7031183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7031887Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7032329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7032773Z return func(*args, **kwargs) 2025-10-10T00:59:12.7033234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7033710Z self_outputs = self.self( 2025-10-10T00:59:12.7034120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7034572Z return func(*args, **kwargs) 2025-10-10T00:59:12.7035033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7035512Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7035673Z 2025-10-10T00:59:12.7035790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7036196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7036560Z res = mod(**inputs) 2025-10-10T00:59:12.7037006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7037494Z outputs = self.bert( 2025-10-10T00:59:12.7037940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7038420Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7038891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7039340Z layer_outputs = layer_module( 2025-10-10T00:59:12.7039698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7040067Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7040456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7040842Z return func(*args, **kwargs) 2025-10-10T00:59:12.7041256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7041702Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7042094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7042472Z return func(*args, **kwargs) 2025-10-10T00:59:12.7042887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7043316Z self_outputs = self.self( 2025-10-10T00:59:12.7043753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7044130Z return func(*args, **kwargs) 2025-10-10T00:59:12.7044541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7044993Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7045142Z 2025-10-10T00:59:12.7045252Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7045626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7045964Z res = mod(**inputs) 2025-10-10T00:59:12.7046417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7046935Z outputs = self.bert( 2025-10-10T00:59:12.7047353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7047797Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7048239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7048673Z layer_outputs = layer_module( 2025-10-10T00:59:12.7049046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7049433Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7049838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7050244Z return func(*args, **kwargs) 2025-10-10T00:59:12.7050668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7051107Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7051488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7051859Z return func(*args, **kwargs) 2025-10-10T00:59:12.7052263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7052668Z self_outputs = self.self( 2025-10-10T00:59:12.7053027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7053396Z return func(*args, **kwargs) 2025-10-10T00:59:12.7053802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7054231Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7054367Z 2025-10-10T00:59:12.7054452Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7054670Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7054914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7055280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7055596Z res = mod(**inputs) 2025-10-10T00:59:12.7055998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7056416Z outputs = self.bert( 2025-10-10T00:59:12.7056809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7057231Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7057637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7058079Z layer_outputs = layer_module( 2025-10-10T00:59:12.7058427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7058789Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7059175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7059546Z return func(*args, **kwargs) 2025-10-10T00:59:12.7059951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7060377Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7060774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7061163Z return func(*args, **kwargs) 2025-10-10T00:59:12.7061573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7062052Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7062536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7062976Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7063118Z 2025-10-10T00:59:12.7063224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7063607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7063951Z res = mod(**inputs) 2025-10-10T00:59:12.7064381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7064827Z outputs = self.bert( 2025-10-10T00:59:12.7065256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7065713Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7066164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7066707Z layer_outputs = layer_module( 2025-10-10T00:59:12.7067091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7067497Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7067927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7068335Z return func(*args, **kwargs) 2025-10-10T00:59:12.7068743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7069171Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7069576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7069978Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7070438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7070925Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7071378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7071818Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7071969Z 2025-10-10T00:59:12.7072075Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7072450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7072784Z res = mod(**inputs) 2025-10-10T00:59:12.7073180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7073597Z outputs = self.bert( 2025-10-10T00:59:12.7074029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7074467Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7074878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7075296Z layer_outputs = layer_module( 2025-10-10T00:59:12.7075681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7076037Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7076416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7076779Z return func(*args, **kwargs) 2025-10-10T00:59:12.7077186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7077614Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7078014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7078406Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7078843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7079321Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7079774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7080242Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7080624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7080973Z return self.act(input) 2025-10-10T00:59:12.7081105Z 2025-10-10T00:59:12.7081209Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7081570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7081891Z res = mod(**inputs) 2025-10-10T00:59:12.7082283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7082709Z outputs = self.bert( 2025-10-10T00:59:12.7083134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7083588Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7084039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7084470Z layer_outputs = layer_module( 2025-10-10T00:59:12.7084834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7085195Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7085575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7085939Z return func(*args, **kwargs) 2025-10-10T00:59:12.7086361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7086807Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7087241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7087642Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7088124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7088634Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7089123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7089566Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7089721Z 2025-10-10T00:59:12.7089845Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7090199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7090528Z res = mod(**inputs) 2025-10-10T00:59:12.7090930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7091352Z outputs = self.bert( 2025-10-10T00:59:12.7091757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7092178Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7092600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7093021Z layer_outputs = layer_module( 2025-10-10T00:59:12.7093380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7093752Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7094148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7094536Z return func(*args, **kwargs) 2025-10-10T00:59:12.7094961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7095413Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7095819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7096227Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7096687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7097207Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7097700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7098134Z return input_tensor + hidden_states 2025-10-10T00:59:12.7098280Z 2025-10-10T00:59:12.7098390Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7098765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7099099Z res = mod(**inputs) 2025-10-10T00:59:12.7099511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7099937Z outputs = self.bert( 2025-10-10T00:59:12.7100346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7100797Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7101252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7101730Z layer_outputs = layer_module( 2025-10-10T00:59:12.7102111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7102510Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7102975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7103383Z return func(*args, **kwargs) 2025-10-10T00:59:12.7103833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7104303Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7104782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7105194Z return func(*args, **kwargs) 2025-10-10T00:59:12.7105647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7106107Z self_outputs = self.self( 2025-10-10T00:59:12.7106597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7107022Z return func(*args, **kwargs) 2025-10-10T00:59:12.7107486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7107952Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7108102Z 2025-10-10T00:59:12.7108217Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7108610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7108973Z res = mod(**inputs) 2025-10-10T00:59:12.7109404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7109847Z outputs = self.bert( 2025-10-10T00:59:12.7110277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7110737Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7111189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7111642Z layer_outputs = layer_module( 2025-10-10T00:59:12.7112011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7112408Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7112816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7113217Z return func(*args, **kwargs) 2025-10-10T00:59:12.7113650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7114119Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7114534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7114929Z return func(*args, **kwargs) 2025-10-10T00:59:12.7115365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7115813Z self_outputs = self.self( 2025-10-10T00:59:12.7116202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7116599Z return func(*args, **kwargs) 2025-10-10T00:59:12.7117034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7117524Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7117670Z 2025-10-10T00:59:12.7117787Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7118186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7118564Z res = mod(**inputs) 2025-10-10T00:59:12.7119009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7119474Z outputs = self.bert( 2025-10-10T00:59:12.7119924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7120409Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7120874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7121346Z layer_outputs = layer_module( 2025-10-10T00:59:12.7121718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7122112Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7122526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7122928Z return func(*args, **kwargs) 2025-10-10T00:59:12.7123370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7123835Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7124268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7124675Z return func(*args, **kwargs) 2025-10-10T00:59:12.7125125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7125574Z self_outputs = self.self( 2025-10-10T00:59:12.7125958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7126360Z return func(*args, **kwargs) 2025-10-10T00:59:12.7126799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7127258Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7127403Z 2025-10-10T00:59:12.7127494Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7127733Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7127992Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7128381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7128730Z res = mod(**inputs) 2025-10-10T00:59:12.7129153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7129603Z outputs = self.bert( 2025-10-10T00:59:12.7130031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7130485Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7130929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7131385Z layer_outputs = layer_module( 2025-10-10T00:59:12.7131967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7132373Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7132839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7133234Z return func(*args, **kwargs) 2025-10-10T00:59:12.7133676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7134170Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7134588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7134988Z return func(*args, **kwargs) 2025-10-10T00:59:12.7135446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7135987Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7136498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7136968Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7137119Z 2025-10-10T00:59:12.7137239Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7137624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7137977Z res = mod(**inputs) 2025-10-10T00:59:12.7138408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7138854Z outputs = self.bert( 2025-10-10T00:59:12.7139278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7139737Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7140185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7140640Z layer_outputs = layer_module( 2025-10-10T00:59:12.7141016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7141403Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7141814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7142216Z return func(*args, **kwargs) 2025-10-10T00:59:12.7142659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7143127Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7143559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7143993Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7144491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7145033Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7145533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7146002Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7146162Z 2025-10-10T00:59:12.7146279Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7146738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7147103Z res = mod(**inputs) 2025-10-10T00:59:12.7147546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7148042Z outputs = self.bert( 2025-10-10T00:59:12.7148487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7148959Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7149442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7149901Z layer_outputs = layer_module( 2025-10-10T00:59:12.7150288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7150691Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7151132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7151567Z return func(*args, **kwargs) 2025-10-10T00:59:12.7152015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7152499Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7152948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7153384Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7153874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7154406Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7154905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7155423Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7155848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7156225Z return self.act(input) 2025-10-10T00:59:12.7156365Z 2025-10-10T00:59:12.7156477Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7156867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7157216Z res = mod(**inputs) 2025-10-10T00:59:12.7157642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7158087Z outputs = self.bert( 2025-10-10T00:59:12.7158518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7158978Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7159429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7159885Z layer_outputs = layer_module( 2025-10-10T00:59:12.7160256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7160651Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7161068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7161474Z return func(*args, **kwargs) 2025-10-10T00:59:12.7161915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7162388Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7162831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7163257Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7163740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7164298Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7164825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7165332Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7165483Z 2025-10-10T00:59:12.7165606Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7166024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7166390Z res = mod(**inputs) 2025-10-10T00:59:12.7166862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7167281Z outputs = self.bert( 2025-10-10T00:59:12.7167686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7168118Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7168537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7168963Z layer_outputs = layer_module( 2025-10-10T00:59:12.7169342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7169745Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7170154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7170575Z return func(*args, **kwargs) 2025-10-10T00:59:12.7171023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7171488Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7171877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7172247Z return func(*args, **kwargs) 2025-10-10T00:59:12.7172671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7173089Z self_outputs = self.self( 2025-10-10T00:59:12.7173446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7173813Z return func(*args, **kwargs) 2025-10-10T00:59:12.7174234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7174688Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7174835Z 2025-10-10T00:59:12.7174942Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7175307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7175650Z res = mod(**inputs) 2025-10-10T00:59:12.7176073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7176496Z outputs = self.bert( 2025-10-10T00:59:12.7176899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7177345Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7180485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7180929Z layer_outputs = layer_module( 2025-10-10T00:59:12.7181298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7181678Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7182066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7182450Z return func(*args, **kwargs) 2025-10-10T00:59:12.7182903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7183352Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7183741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7184207Z return func(*args, **kwargs) 2025-10-10T00:59:12.7184667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7185137Z self_outputs = self.self( 2025-10-10T00:59:12.7185535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7185954Z return func(*args, **kwargs) 2025-10-10T00:59:12.7186477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7186956Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7187110Z 2025-10-10T00:59:12.7187237Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7187636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7188002Z res = mod(**inputs) 2025-10-10T00:59:12.7188448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7188887Z outputs = self.bert( 2025-10-10T00:59:12.7189299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7189728Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7190167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7190604Z layer_outputs = layer_module( 2025-10-10T00:59:12.7190967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7191345Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7191732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7192117Z return func(*args, **kwargs) 2025-10-10T00:59:12.7192540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7192982Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7193371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7193756Z return func(*args, **kwargs) 2025-10-10T00:59:12.7194177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7194609Z self_outputs = self.self( 2025-10-10T00:59:12.7194977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7195346Z return func(*args, **kwargs) 2025-10-10T00:59:12.7195856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7196330Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7196479Z 2025-10-10T00:59:12.7196576Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7196812Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7197066Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7197458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7197830Z res = mod(**inputs) 2025-10-10T00:59:12.7198261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7198714Z outputs = self.bert( 2025-10-10T00:59:12.7199164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7199636Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7200090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7200552Z layer_outputs = layer_module( 2025-10-10T00:59:12.7200925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7201329Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7201742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7202147Z return func(*args, **kwargs) 2025-10-10T00:59:12.7202590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7203062Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7203478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7203877Z return func(*args, **kwargs) 2025-10-10T00:59:12.7204322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7204829Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7205351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7205818Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7205969Z 2025-10-10T00:59:12.7206093Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7206485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7206829Z res = mod(**inputs) 2025-10-10T00:59:12.7207265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7207714Z outputs = self.bert( 2025-10-10T00:59:12.7208140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7208595Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7209041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7209492Z layer_outputs = layer_module( 2025-10-10T00:59:12.7209873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7210265Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7210668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7211115Z return func(*args, **kwargs) 2025-10-10T00:59:12.7211555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7212021Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7212455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7212877Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7213379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7213901Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7214384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7214884Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7215032Z 2025-10-10T00:59:12.7215144Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7215531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7215877Z res = mod(**inputs) 2025-10-10T00:59:12.7216305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7216755Z outputs = self.bert( 2025-10-10T00:59:12.7217174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7217624Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7218075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7218526Z layer_outputs = layer_module( 2025-10-10T00:59:12.7218895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7219294Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7219703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7220111Z return func(*args, **kwargs) 2025-10-10T00:59:12.7220560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7221017Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7221452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7221874Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7222360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7222880Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7223371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7223881Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7224302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7224684Z return self.act(input) 2025-10-10T00:59:12.7224810Z 2025-10-10T00:59:12.7224935Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7225327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7225684Z res = mod(**inputs) 2025-10-10T00:59:12.7226125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7226718Z outputs = self.bert( 2025-10-10T00:59:12.7227156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7227628Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7228094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7228587Z layer_outputs = layer_module( 2025-10-10T00:59:12.7228987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7229386Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7229817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7230276Z return func(*args, **kwargs) 2025-10-10T00:59:12.7230738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7231228Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7231858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7232311Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7232821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7233385Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7233915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7234392Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7234558Z 2025-10-10T00:59:12.7234678Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7235083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7235450Z res = mod(**inputs) 2025-10-10T00:59:12.7235887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7236351Z outputs = self.bert( 2025-10-10T00:59:12.7236795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7237246Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7237671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7238092Z layer_outputs = layer_module( 2025-10-10T00:59:12.7238447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7238807Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7239190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7239567Z return func(*args, **kwargs) 2025-10-10T00:59:12.7239978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7240430Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7240827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7241217Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7241655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7242214Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7242691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7243114Z return input_tensor + hidden_states 2025-10-10T00:59:12.7243248Z 2025-10-10T00:59:12.7243366Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7243769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7244102Z res = mod(**inputs) 2025-10-10T00:59:12.7244515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7244945Z outputs = self.bert( 2025-10-10T00:59:12.7245388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7245827Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7246245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7246662Z layer_outputs = layer_module( 2025-10-10T00:59:12.7247012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7247375Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7247752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7248126Z return func(*args, **kwargs) 2025-10-10T00:59:12.7248538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7248973Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7249357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7249730Z return func(*args, **kwargs) 2025-10-10T00:59:12.7250140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7250559Z self_outputs = self.self( 2025-10-10T00:59:12.7250922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7250991Z return func(*args, **kwargs) 2025-10-10T00:59:12.7251276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7251366Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7251369Z 2025-10-10T00:59:12.7251475Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7251684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7251748Z res = mod(**inputs) 2025-10-10T00:59:12.7252045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7252114Z outputs = self.bert( 2025-10-10T00:59:12.7252400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7252483Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7252765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7252844Z layer_outputs = layer_module( 2025-10-10T00:59:12.7253068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7253169Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7253414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7253485Z return func(*args, **kwargs) 2025-10-10T00:59:12.7253822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7253906Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7254167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7254239Z return func(*args, **kwargs) 2025-10-10T00:59:12.7254580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7254658Z self_outputs = self.self( 2025-10-10T00:59:12.7254922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7254996Z return func(*args, **kwargs) 2025-10-10T00:59:12.7255274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7255351Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7255355Z 2025-10-10T00:59:12.7255464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7255661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7255733Z res = mod(**inputs) 2025-10-10T00:59:12.7256020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7256084Z outputs = self.bert( 2025-10-10T00:59:12.7256378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7256453Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7256743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7256813Z layer_outputs = layer_module( 2025-10-10T00:59:12.7257040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7257118Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7257357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7257433Z return func(*args, **kwargs) 2025-10-10T00:59:12.7257717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7257810Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7258047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7258115Z return func(*args, **kwargs) 2025-10-10T00:59:12.7258407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7258477Z self_outputs = self.self( 2025-10-10T00:59:12.7258722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7258791Z return func(*args, **kwargs) 2025-10-10T00:59:12.7259075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7259154Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7259174Z 2025-10-10T00:59:12.7259258Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7259344Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7259449Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7259653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7259719Z res = mod(**inputs) 2025-10-10T00:59:12.7260008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7260084Z outputs = self.bert( 2025-10-10T00:59:12.7260383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7260466Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7260749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7260865Z layer_outputs = layer_module( 2025-10-10T00:59:12.7261094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7261173Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7261417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7261485Z return func(*args, **kwargs) 2025-10-10T00:59:12.7261799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7261889Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7262146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7262226Z return func(*args, **kwargs) 2025-10-10T00:59:12.7262541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7262690Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7263000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7263093Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7263106Z 2025-10-10T00:59:12.7263219Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7263433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7263509Z res = mod(**inputs) 2025-10-10T00:59:12.7263827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7263907Z outputs = self.bert( 2025-10-10T00:59:12.7264218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7264299Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7264616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7264693Z layer_outputs = layer_module( 2025-10-10T00:59:12.7264938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7265024Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7265281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7265364Z return func(*args, **kwargs) 2025-10-10T00:59:12.7265679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7265802Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7266080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7266171Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7266591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7266734Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7267084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7267179Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7267183Z 2025-10-10T00:59:12.7267317Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7267552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7267646Z res = mod(**inputs) 2025-10-10T00:59:12.7267981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7268048Z outputs = self.bert( 2025-10-10T00:59:12.7268332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7268404Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7268687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7268757Z layer_outputs = layer_module( 2025-10-10T00:59:12.7268970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7269054Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7269291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7269367Z return func(*args, **kwargs) 2025-10-10T00:59:12.7269640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7269720Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7269979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7270056Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7270366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7270467Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7270746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7270858Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7271064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7271141Z return self.act(input) 2025-10-10T00:59:12.7271145Z 2025-10-10T00:59:12.7271245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7271444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7271508Z res = mod(**inputs) 2025-10-10T00:59:12.7271785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7271857Z outputs = self.bert( 2025-10-10T00:59:12.7272132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7272233Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7272511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7272581Z layer_outputs = layer_module( 2025-10-10T00:59:12.7272803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7272881Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7273148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7273219Z return func(*args, **kwargs) 2025-10-10T00:59:12.7273506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7273591Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7273886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7273971Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7274284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7274427Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7274720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7274811Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7274815Z 2025-10-10T00:59:12.7274921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7275125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7275203Z res = mod(**inputs) 2025-10-10T00:59:12.7275506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7275579Z outputs = self.bert( 2025-10-10T00:59:12.7275862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7275936Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7276235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7276306Z layer_outputs = layer_module( 2025-10-10T00:59:12.7276532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7276610Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7276851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7276930Z return func(*args, **kwargs) 2025-10-10T00:59:12.7277212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7277303Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7277544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7277621Z return func(*args, **kwargs) 2025-10-10T00:59:12.7277906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7277976Z self_outputs = self.self( 2025-10-10T00:59:12.7278220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7278307Z return func(*args, **kwargs) 2025-10-10T00:59:12.7278599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7278681Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7278684Z 2025-10-10T00:59:12.7278785Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7278987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7279052Z res = mod(**inputs) 2025-10-10T00:59:12.7279361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7279430Z outputs = self.bert( 2025-10-10T00:59:12.7279718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7279792Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7280112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7280194Z layer_outputs = layer_module( 2025-10-10T00:59:12.7280413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7280499Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7280735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7280805Z return func(*args, **kwargs) 2025-10-10T00:59:12.7281095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7281178Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7281424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7281496Z return func(*args, **kwargs) 2025-10-10T00:59:12.7281777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7281855Z self_outputs = self.self( 2025-10-10T00:59:12.7282091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7282168Z return func(*args, **kwargs) 2025-10-10T00:59:12.7282451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7282540Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7282543Z 2025-10-10T00:59:12.7282645Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7282839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7282916Z res = mod(**inputs) 2025-10-10T00:59:12.7283200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7283274Z outputs = self.bert( 2025-10-10T00:59:12.7283557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7283628Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7283924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7283996Z layer_outputs = layer_module( 2025-10-10T00:59:12.7284226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7284306Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7284576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7284647Z return func(*args, **kwargs) 2025-10-10T00:59:12.7284940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7285043Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7285282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7285373Z return func(*args, **kwargs) 2025-10-10T00:59:12.7285655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7285723Z self_outputs = self.self( 2025-10-10T00:59:12.7285966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7286083Z return func(*args, **kwargs) 2025-10-10T00:59:12.7286371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7286451Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7286454Z 2025-10-10T00:59:12.7286543Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7286622Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7286722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7286926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7286990Z res = mod(**inputs) 2025-10-10T00:59:12.7287281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7287348Z outputs = self.bert( 2025-10-10T00:59:12.7287632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7287715Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7287998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7288078Z layer_outputs = layer_module( 2025-10-10T00:59:12.7288297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7288378Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7288624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7288694Z return func(*args, **kwargs) 2025-10-10T00:59:12.7288992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7289077Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7289316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7289383Z return func(*args, **kwargs) 2025-10-10T00:59:12.7289658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7289791Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7290066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7290155Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7290159Z 2025-10-10T00:59:12.7290258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7290446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7290537Z res = mod(**inputs) 2025-10-10T00:59:12.7290814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7290886Z outputs = self.bert( 2025-10-10T00:59:12.7291163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7291241Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7291533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7291604Z layer_outputs = layer_module( 2025-10-10T00:59:12.7291823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7291899Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7292177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7292246Z return func(*args, **kwargs) 2025-10-10T00:59:12.7292525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7292616Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7292872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7292958Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7293272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7293380Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7293704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7293789Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7293793Z 2025-10-10T00:59:12.7293900Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7294095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7294167Z res = mod(**inputs) 2025-10-10T00:59:12.7294455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7294525Z outputs = self.bert( 2025-10-10T00:59:12.7294815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7294888Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7295182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7295256Z layer_outputs = layer_module( 2025-10-10T00:59:12.7295480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7295568Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7295801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7295876Z return func(*args, **kwargs) 2025-10-10T00:59:12.7296150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7296239Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7296489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7296563Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7296895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7296996Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7297327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7297439Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7297671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7297743Z return self.act(input) 2025-10-10T00:59:12.7297747Z 2025-10-10T00:59:12.7297846Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7298055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7298120Z res = mod(**inputs) 2025-10-10T00:59:12.7298434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7298502Z outputs = self.bert( 2025-10-10T00:59:12.7298774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7298855Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7299130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7299208Z layer_outputs = layer_module( 2025-10-10T00:59:12.7299427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7299507Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7299757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7299830Z return func(*args, **kwargs) 2025-10-10T00:59:12.7300117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7300201Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7300460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7300534Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7300847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7300988Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7301275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7301370Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7301374Z 2025-10-10T00:59:12.7301477Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7301684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7301748Z res = mod(**inputs) 2025-10-10T00:59:12.7302042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7302125Z outputs = self.bert( 2025-10-10T00:59:12.7302440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7302527Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7302839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7302941Z layer_outputs = layer_module( 2025-10-10T00:59:12.7303186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7303273Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7303540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7303614Z return func(*args, **kwargs) 2025-10-10T00:59:12.7303942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7304041Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7304319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7304409Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7304772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7304938Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7305257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7305341Z return input_tensor + hidden_states 2025-10-10T00:59:12.7305345Z 2025-10-10T00:59:12.7305465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7305678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7305756Z res = mod(**inputs) 2025-10-10T00:59:12.7306064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7306136Z outputs = self.bert( 2025-10-10T00:59:12.7306523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7306609Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7306928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7307008Z layer_outputs = layer_module( 2025-10-10T00:59:12.7307258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7307347Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7307606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7307691Z return func(*args, **kwargs) 2025-10-10T00:59:12.7308008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7308114Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7308378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7308462Z return func(*args, **kwargs) 2025-10-10T00:59:12.7308786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7308865Z self_outputs = self.self( 2025-10-10T00:59:12.7309134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7309208Z return func(*args, **kwargs) 2025-10-10T00:59:12.7309522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7309610Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7309641Z 2025-10-10T00:59:12.7309757Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7309977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7310047Z res = mod(**inputs) 2025-10-10T00:59:12.7310363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7310435Z outputs = self.bert( 2025-10-10T00:59:12.7310785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7310876Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7311182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7311267Z layer_outputs = layer_module( 2025-10-10T00:59:12.7311533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7311647Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7311908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7311983Z return func(*args, **kwargs) 2025-10-10T00:59:12.7312298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7312391Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7312657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7312732Z return func(*args, **kwargs) 2025-10-10T00:59:12.7313041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7313130Z self_outputs = self.self( 2025-10-10T00:59:12.7313389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7313471Z return func(*args, **kwargs) 2025-10-10T00:59:12.7313779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7313866Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7313878Z 2025-10-10T00:59:12.7313989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7314203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7314280Z res = mod(**inputs) 2025-10-10T00:59:12.7314592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7314672Z outputs = self.bert( 2025-10-10T00:59:12.7314986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7315064Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7315379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7315456Z layer_outputs = layer_module( 2025-10-10T00:59:12.7315700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7315788Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7316043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7316124Z return func(*args, **kwargs) 2025-10-10T00:59:12.7316433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7316553Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7316813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7316892Z return func(*args, **kwargs) 2025-10-10T00:59:12.7317200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7317274Z self_outputs = self.self( 2025-10-10T00:59:12.7317560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7317634Z return func(*args, **kwargs) 2025-10-10T00:59:12.7317948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7318039Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7318078Z 2025-10-10T00:59:12.7318169Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7318262Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7318373Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7318591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7318661Z res = mod(**inputs) 2025-10-10T00:59:12.7318971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7319048Z outputs = self.bert( 2025-10-10T00:59:12.7319338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7319419Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7319710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7319793Z layer_outputs = layer_module( 2025-10-10T00:59:12.7320015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7320096Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7320347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7320419Z return func(*args, **kwargs) 2025-10-10T00:59:12.7320718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7320804Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7321046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7321124Z return func(*args, **kwargs) 2025-10-10T00:59:12.7321416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7321554Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7321843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7321937Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7321940Z 2025-10-10T00:59:12.7322046Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7322247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7322321Z res = mod(**inputs) 2025-10-10T00:59:12.7322624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7322730Z outputs = self.bert( 2025-10-10T00:59:12.7323047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7323128Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7323461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7323540Z layer_outputs = layer_module( 2025-10-10T00:59:12.7323795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7323901Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7324170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7324255Z return func(*args, **kwargs) 2025-10-10T00:59:12.7324601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7324724Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7325004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7325094Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7325433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7325547Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7325865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7325954Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7325958Z 2025-10-10T00:59:12.7326076Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7326293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7326369Z res = mod(**inputs) 2025-10-10T00:59:12.7326692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7326765Z outputs = self.bert( 2025-10-10T00:59:12.7327087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7327167Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7327491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7327568Z layer_outputs = layer_module( 2025-10-10T00:59:12.7327808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7327906Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7328165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7328252Z return func(*args, **kwargs) 2025-10-10T00:59:12.7328569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7328661Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7328949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7329031Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7329378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7329489Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7329850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7329965Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7330183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7330263Z return self.act(input) 2025-10-10T00:59:12.7330267Z 2025-10-10T00:59:12.7330371Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7330594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7330664Z res = mod(**inputs) 2025-10-10T00:59:12.7330954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7331032Z outputs = self.bert( 2025-10-10T00:59:12.7331347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7331624Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7331934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7332020Z layer_outputs = layer_module( 2025-10-10T00:59:12.7332260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7332351Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7332632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7332705Z return func(*args, **kwargs) 2025-10-10T00:59:12.7333005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7333097Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7333366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7333454Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7333782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7333926Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7334226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7334321Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7334325Z 2025-10-10T00:59:12.7334432Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7334639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7334718Z res = mod(**inputs) 2025-10-10T00:59:12.7335021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7335100Z outputs = self.bert( 2025-10-10T00:59:12.7335397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7335473Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7335780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7335854Z layer_outputs = layer_module( 2025-10-10T00:59:12.7336092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7336174Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7336480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7336553Z return func(*args, **kwargs) 2025-10-10T00:59:12.7336846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7336939Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7337183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7337287Z return func(*args, **kwargs) 2025-10-10T00:59:12.7337579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7337653Z self_outputs = self.self( 2025-10-10T00:59:12.7337903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7338714Z return func(*args, **kwargs) 2025-10-10T00:59:12.7339021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7339108Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7339111Z 2025-10-10T00:59:12.7339223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7339425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7339494Z res = mod(**inputs) 2025-10-10T00:59:12.7339797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7339866Z outputs = self.bert( 2025-10-10T00:59:12.7340166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7340245Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7340536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7340617Z layer_outputs = layer_module( 2025-10-10T00:59:12.7340844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7340935Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7341182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7341252Z return func(*args, **kwargs) 2025-10-10T00:59:12.7341551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7341635Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7341890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7341963Z return func(*args, **kwargs) 2025-10-10T00:59:12.7342259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7342331Z self_outputs = self.self( 2025-10-10T00:59:12.7342574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7342651Z return func(*args, **kwargs) 2025-10-10T00:59:12.7342947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7343034Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7343037Z 2025-10-10T00:59:12.7343143Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7343362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7343438Z res = mod(**inputs) 2025-10-10T00:59:12.7343731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7343805Z outputs = self.bert( 2025-10-10T00:59:12.7344097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7344179Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7344488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7344567Z layer_outputs = layer_module( 2025-10-10T00:59:12.7344811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7344898Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7345205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7345281Z return func(*args, **kwargs) 2025-10-10T00:59:12.7345596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7345690Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7345948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7346030Z return func(*args, **kwargs) 2025-10-10T00:59:12.7346392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7346484Z self_outputs = self.self( 2025-10-10T00:59:12.7346750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7346828Z return func(*args, **kwargs) 2025-10-10T00:59:12.7347152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7347244Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7347248Z 2025-10-10T00:59:12.7347346Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7347437Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7347552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7347792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7347857Z res = mod(**inputs) 2025-10-10T00:59:12.7348160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7348231Z outputs = self.bert( 2025-10-10T00:59:12.7348525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7348608Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7348904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7348983Z layer_outputs = layer_module( 2025-10-10T00:59:12.7349207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7349295Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7349540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7349609Z return func(*args, **kwargs) 2025-10-10T00:59:12.7349912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7350020Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7350261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7350330Z return func(*args, **kwargs) 2025-10-10T00:59:12.7350611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7350744Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7351041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7351133Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7351137Z 2025-10-10T00:59:12.7351237Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7351460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7351547Z res = mod(**inputs) 2025-10-10T00:59:12.7351838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7351913Z outputs = self.bert( 2025-10-10T00:59:12.7352198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7352276Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7352561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7352635Z layer_outputs = layer_module( 2025-10-10T00:59:12.7352860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7352940Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7353188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7353257Z return func(*args, **kwargs) 2025-10-10T00:59:12.7353540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7353632Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7353893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7353975Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7354287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7354399Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7354687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7354772Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7354775Z 2025-10-10T00:59:12.7354883Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7355080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7355152Z res = mod(**inputs) 2025-10-10T00:59:12.7355441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7355511Z outputs = self.bert( 2025-10-10T00:59:12.7355802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7355876Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7356194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7356268Z layer_outputs = layer_module( 2025-10-10T00:59:12.7356494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7356572Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7356812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7356891Z return func(*args, **kwargs) 2025-10-10T00:59:12.7357200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7357298Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7357550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7357670Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7357979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7358078Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7358361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7358469Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7358684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7358752Z return self.act(input) 2025-10-10T00:59:12.7358756Z 2025-10-10T00:59:12.7358854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7359052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7359119Z res = mod(**inputs) 2025-10-10T00:59:12.7359408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7359473Z outputs = self.bert( 2025-10-10T00:59:12.7359789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7359870Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7360156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7360236Z layer_outputs = layer_module( 2025-10-10T00:59:12.7360456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7360540Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7360780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7360851Z return func(*args, **kwargs) 2025-10-10T00:59:12.7361140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7361222Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7361484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7361562Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7361880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7362015Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7362292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7362405Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7362408Z 2025-10-10T00:59:12.7362508Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7362706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7362769Z res = mod(**inputs) 2025-10-10T00:59:12.7363050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7363139Z outputs = self.bert( 2025-10-10T00:59:12.7363418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7363501Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7363812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7363904Z layer_outputs = layer_module( 2025-10-10T00:59:12.7364138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7364216Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7364456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7364523Z return func(*args, **kwargs) 2025-10-10T00:59:12.7364807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7364889Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7365136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7365216Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7365533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7365671Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7365964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7366041Z return input_tensor + hidden_states 2025-10-10T00:59:12.7366051Z 2025-10-10T00:59:12.7366150Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7366340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7366411Z res = mod(**inputs) 2025-10-10T00:59:12.7366693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7366771Z outputs = self.bert( 2025-10-10T00:59:12.7367056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7367131Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7367421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7367492Z layer_outputs = layer_module( 2025-10-10T00:59:12.7367719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7367799Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7368035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7368112Z return func(*args, **kwargs) 2025-10-10T00:59:12.7368399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7368511Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7368744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7368822Z return func(*args, **kwargs) 2025-10-10T00:59:12.7369106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7369176Z self_outputs = self.self( 2025-10-10T00:59:12.7369437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7369507Z return func(*args, **kwargs) 2025-10-10T00:59:12.7369795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7369916Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7369920Z 2025-10-10T00:59:12.7370023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7370225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7370290Z res = mod(**inputs) 2025-10-10T00:59:12.7370584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7370651Z outputs = self.bert( 2025-10-10T00:59:12.7370935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7371017Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7371300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7371381Z layer_outputs = layer_module( 2025-10-10T00:59:12.7371602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7371689Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7371927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7371996Z return func(*args, **kwargs) 2025-10-10T00:59:12.7372287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7372372Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7372615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7372682Z return func(*args, **kwargs) 2025-10-10T00:59:12.7372967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7373051Z self_outputs = self.self( 2025-10-10T00:59:12.7373289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7373363Z return func(*args, **kwargs) 2025-10-10T00:59:12.7373649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7373735Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7373739Z 2025-10-10T00:59:12.7373842Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7374037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7374111Z res = mod(**inputs) 2025-10-10T00:59:12.7374395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7374496Z outputs = self.bert( 2025-10-10T00:59:12.7374777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7374849Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7375136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7375209Z layer_outputs = layer_module( 2025-10-10T00:59:12.7375449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7375529Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7375773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7375842Z return func(*args, **kwargs) 2025-10-10T00:59:12.7376144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7376257Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7376492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7376568Z return func(*args, **kwargs) 2025-10-10T00:59:12.7376851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7376922Z self_outputs = self.self( 2025-10-10T00:59:12.7377165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7377233Z return func(*args, **kwargs) 2025-10-10T00:59:12.7377522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7377607Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7377610Z 2025-10-10T00:59:12.7377696Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7377776Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7377879Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7378082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7378146Z res = mod(**inputs) 2025-10-10T00:59:12.7378441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7378512Z outputs = self.bert( 2025-10-10T00:59:12.7378815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7378901Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7379212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7379295Z layer_outputs = layer_module( 2025-10-10T00:59:12.7379517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7379596Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7379846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7379918Z return func(*args, **kwargs) 2025-10-10T00:59:12.7380213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7380298Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7380555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7380662Z return func(*args, **kwargs) 2025-10-10T00:59:12.7380967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7381112Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7381417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7381515Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7381519Z 2025-10-10T00:59:12.7381646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7381860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7381940Z res = mod(**inputs) 2025-10-10T00:59:12.7382252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7382381Z outputs = self.bert( 2025-10-10T00:59:12.7382686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7382765Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7383090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7383168Z layer_outputs = layer_module( 2025-10-10T00:59:12.7383417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7383501Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7383767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7383843Z return func(*args, **kwargs) 2025-10-10T00:59:12.7384151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7384251Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7384529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7384620Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7384969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7385086Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7385411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7385505Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7385509Z 2025-10-10T00:59:12.7385634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7385853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7385934Z res = mod(**inputs) 2025-10-10T00:59:12.7386262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7386533Z outputs = self.bert( 2025-10-10T00:59:12.7386886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7386973Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7387302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7387391Z layer_outputs = layer_module( 2025-10-10T00:59:12.7387621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7387740Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7387986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7388067Z return func(*args, **kwargs) 2025-10-10T00:59:12.7388364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7388458Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7388734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7388814Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7389132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7389256Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7389564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7389677Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7389889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7389971Z return self.act(input) 2025-10-10T00:59:12.7389975Z 2025-10-10T00:59:12.7390079Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7390288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7390357Z res = mod(**inputs) 2025-10-10T00:59:12.7390658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7390728Z outputs = self.bert( 2025-10-10T00:59:12.7391019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7391102Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7391396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7391474Z layer_outputs = layer_module( 2025-10-10T00:59:12.7391691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7391771Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7392015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7392085Z return func(*args, **kwargs) 2025-10-10T00:59:12.7392378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7392463Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7392724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7392800Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7393112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7393255Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7393543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7393636Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7393640Z 2025-10-10T00:59:12.7393743Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7393965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7394039Z res = mod(**inputs) 2025-10-10T00:59:12.7394336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7394417Z outputs = self.bert( 2025-10-10T00:59:12.7394724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7394810Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7395135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7395215Z layer_outputs = layer_module( 2025-10-10T00:59:12.7395460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7395602Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7395853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7395924Z return func(*args, **kwargs) 2025-10-10T00:59:12.7396209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7396304Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7396547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7396625Z return func(*args, **kwargs) 2025-10-10T00:59:12.7396911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7396992Z self_outputs = self.self( 2025-10-10T00:59:12.7397234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7397308Z return func(*args, **kwargs) 2025-10-10T00:59:12.7397620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7397709Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7397713Z 2025-10-10T00:59:12.7397830Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7398041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7398112Z res = mod(**inputs) 2025-10-10T00:59:12.7398425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7398497Z outputs = self.bert( 2025-10-10T00:59:12.7398809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7398890Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7399190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7399275Z layer_outputs = layer_module( 2025-10-10T00:59:12.7399511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7399604Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7399863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7399945Z return func(*args, **kwargs) 2025-10-10T00:59:12.7400247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7400359Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7400636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7400707Z return func(*args, **kwargs) 2025-10-10T00:59:12.7401003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7401077Z self_outputs = self.self( 2025-10-10T00:59:12.7401320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7401413Z return func(*args, **kwargs) 2025-10-10T00:59:12.7401707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7401795Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7401799Z 2025-10-10T00:59:12.7401904Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7402188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7402257Z res = mod(**inputs) 2025-10-10T00:59:12.7402552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7402630Z outputs = self.bert( 2025-10-10T00:59:12.7402921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7403007Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7403315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7403394Z layer_outputs = layer_module( 2025-10-10T00:59:12.7403639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7403729Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7403994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7404067Z return func(*args, **kwargs) 2025-10-10T00:59:12.7404387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7404472Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7404717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7404797Z return func(*args, **kwargs) 2025-10-10T00:59:12.7405084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7405163Z self_outputs = self.self( 2025-10-10T00:59:12.7405409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7405479Z return func(*args, **kwargs) 2025-10-10T00:59:12.7405778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7405860Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7405863Z 2025-10-10T00:59:12.7405952Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7406034Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7406141Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7406360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7406431Z res = mod(**inputs) 2025-10-10T00:59:12.7406748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7406847Z outputs = self.bert( 2025-10-10T00:59:12.7407160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7407240Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7407557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7407641Z layer_outputs = layer_module( 2025-10-10T00:59:12.7407903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7407997Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7408257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7408328Z return func(*args, **kwargs) 2025-10-10T00:59:12.7408645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7408750Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7409013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7409090Z return func(*args, **kwargs) 2025-10-10T00:59:12.7409411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7409562Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7409872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7409972Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7409976Z 2025-10-10T00:59:12.7410091Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7410320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7410394Z res = mod(**inputs) 2025-10-10T00:59:12.7410716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7410799Z outputs = self.bert( 2025-10-10T00:59:12.7411110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7411200Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7411519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7411600Z layer_outputs = layer_module( 2025-10-10T00:59:12.7411845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7411937Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7412203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7412281Z return func(*args, **kwargs) 2025-10-10T00:59:12.7412603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7412696Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7412978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7413071Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7413412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7413531Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7413863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7413953Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7413963Z 2025-10-10T00:59:12.7414071Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7414285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7414364Z res = mod(**inputs) 2025-10-10T00:59:12.7414694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7414775Z outputs = self.bert( 2025-10-10T00:59:12.7415080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7415160Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7415508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7415587Z layer_outputs = layer_module( 2025-10-10T00:59:12.7415832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7415916Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7416171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7416255Z return func(*args, **kwargs) 2025-10-10T00:59:12.7416565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7416663Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7416940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7417034Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7417373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7417485Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7417797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7417920Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7418159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7418235Z return self.act(input) 2025-10-10T00:59:12.7418239Z 2025-10-10T00:59:12.7418349Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7418567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7418640Z res = mod(**inputs) 2025-10-10T00:59:12.7418954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7419026Z outputs = self.bert( 2025-10-10T00:59:12.7419340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7419418Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7419724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7419809Z layer_outputs = layer_module( 2025-10-10T00:59:12.7420045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7420158Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7420420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7420496Z return func(*args, **kwargs) 2025-10-10T00:59:12.7420809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7420901Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7421200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7421285Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7421627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7421767Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7422106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7422205Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7422209Z 2025-10-10T00:59:12.7422731Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7422950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7423021Z res = mod(**inputs) 2025-10-10T00:59:12.7423340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7423414Z outputs = self.bert( 2025-10-10T00:59:12.7423719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7423809Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7424119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7424207Z layer_outputs = layer_module( 2025-10-10T00:59:12.7424443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7424527Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7424792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7424866Z return func(*args, **kwargs) 2025-10-10T00:59:12.7425186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7425276Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7425554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7425649Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7425991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7426146Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7426538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7426638Z return input_tensor + hidden_states 2025-10-10T00:59:12.7426643Z 2025-10-10T00:59:12.7426762Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7426985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7427066Z res = mod(**inputs) 2025-10-10T00:59:12.7427400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7427511Z outputs = self.bert( 2025-10-10T00:59:12.7427833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7427921Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7428251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7428330Z layer_outputs = layer_module( 2025-10-10T00:59:12.7428597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7428688Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7428962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7429040Z return func(*args, **kwargs) 2025-10-10T00:59:12.7429393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7429496Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7429762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7429847Z return func(*args, **kwargs) 2025-10-10T00:59:12.7430172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7430253Z self_outputs = self.self( 2025-10-10T00:59:12.7430526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7430601Z return func(*args, **kwargs) 2025-10-10T00:59:12.7430926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7431021Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7431025Z 2025-10-10T00:59:12.7431143Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7431362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7431433Z res = mod(**inputs) 2025-10-10T00:59:12.7431933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7432013Z outputs = self.bert( 2025-10-10T00:59:12.7432345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7432426Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7432745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7432836Z layer_outputs = layer_module( 2025-10-10T00:59:12.7433082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7433178Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7433444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7433531Z return func(*args, **kwargs) 2025-10-10T00:59:12.7433910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7434004Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7434278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7434356Z return func(*args, **kwargs) 2025-10-10T00:59:12.7434747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7434829Z self_outputs = self.self( 2025-10-10T00:59:12.7435092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7435177Z return func(*args, **kwargs) 2025-10-10T00:59:12.7435494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7435615Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7435620Z 2025-10-10T00:59:12.7435736Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7435955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7436035Z res = mod(**inputs) 2025-10-10T00:59:12.7436378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7436491Z outputs = self.bert( 2025-10-10T00:59:12.7436811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7436902Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7437222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7437302Z layer_outputs = layer_module( 2025-10-10T00:59:12.7437559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7437648Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7437926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7438005Z return func(*args, **kwargs) 2025-10-10T00:59:12.7438332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7438433Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7438712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7438795Z return func(*args, **kwargs) 2025-10-10T00:59:12.7439120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7439199Z self_outputs = self.self( 2025-10-10T00:59:12.7439445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7439517Z return func(*args, **kwargs) 2025-10-10T00:59:12.7439822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7439909Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7439912Z 2025-10-10T00:59:12.7440002Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7440085Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7440189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7440399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7440465Z res = mod(**inputs) 2025-10-10T00:59:12.7440771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7440838Z outputs = self.bert( 2025-10-10T00:59:12.7441129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7441227Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7441523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7441604Z layer_outputs = layer_module( 2025-10-10T00:59:12.7441840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7441931Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7442193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7442286Z return func(*args, **kwargs) 2025-10-10T00:59:12.7442594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7442677Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7442944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7443032Z return func(*args, **kwargs) 2025-10-10T00:59:12.7443323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7443459Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7443747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7443842Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7443846Z 2025-10-10T00:59:12.7443951Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7444158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7444224Z res = mod(**inputs) 2025-10-10T00:59:12.7444520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7444598Z outputs = self.bert( 2025-10-10T00:59:12.7444888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7444970Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7445261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7445333Z layer_outputs = layer_module( 2025-10-10T00:59:12.7445567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7445646Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7445896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7445966Z return func(*args, **kwargs) 2025-10-10T00:59:12.7446264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7446351Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7446615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7446701Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7447023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7447134Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7447426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7447511Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7447533Z 2025-10-10T00:59:12.7447646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7447848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7447919Z res = mod(**inputs) 2025-10-10T00:59:12.7448214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7448288Z outputs = self.bert( 2025-10-10T00:59:12.7448594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7448669Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7448963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7449036Z layer_outputs = layer_module( 2025-10-10T00:59:12.7449287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7449387Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7449633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7449711Z return func(*args, **kwargs) 2025-10-10T00:59:12.7450002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7450095Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7450357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7450436Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7450762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7450873Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7451174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7451290Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7451513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7451586Z return self.act(input) 2025-10-10T00:59:12.7451589Z 2025-10-10T00:59:12.7451696Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7451903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7451969Z res = mod(**inputs) 2025-10-10T00:59:12.7452270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7452341Z outputs = self.bert( 2025-10-10T00:59:12.7452631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7452712Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7452998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7453078Z layer_outputs = layer_module( 2025-10-10T00:59:12.7453304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7453392Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7453639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7453709Z return func(*args, **kwargs) 2025-10-10T00:59:12.7453998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7454101Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7454365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7454442Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7454753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7454907Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7455187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7455275Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7455279Z 2025-10-10T00:59:12.7455400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7455624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7455690Z res = mod(**inputs) 2025-10-10T00:59:12.7455974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7456050Z outputs = self.bert( 2025-10-10T00:59:12.7456329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7456409Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7456690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7456760Z layer_outputs = layer_module( 2025-10-10T00:59:12.7456986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7457069Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7457312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7457382Z return func(*args, **kwargs) 2025-10-10T00:59:12.7457673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7457756Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7457993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7458072Z return func(*args, **kwargs) 2025-10-10T00:59:12.7458352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7458431Z self_outputs = self.self( 2025-10-10T00:59:12.7458672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7458740Z return func(*args, **kwargs) 2025-10-10T00:59:12.7459030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7459111Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7459114Z 2025-10-10T00:59:12.7459223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7459419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7459490Z res = mod(**inputs) 2025-10-10T00:59:12.7459775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7459843Z outputs = self.bert( 2025-10-10T00:59:12.7460152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7460225Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7460514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7460585Z layer_outputs = layer_module( 2025-10-10T00:59:12.7460804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7460905Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7461144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7461219Z return func(*args, **kwargs) 2025-10-10T00:59:12.7461511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7461634Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7461880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7461950Z return func(*args, **kwargs) 2025-10-10T00:59:12.7462248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7462320Z self_outputs = self.self( 2025-10-10T00:59:12.7462572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7462643Z return func(*args, **kwargs) 2025-10-10T00:59:12.7462932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7463021Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7463026Z 2025-10-10T00:59:12.7463133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7463339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7463405Z res = mod(**inputs) 2025-10-10T00:59:12.7463701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7463777Z outputs = self.bert( 2025-10-10T00:59:12.7464072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7464153Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7464444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7464524Z layer_outputs = layer_module( 2025-10-10T00:59:12.7464751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7464834Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7465084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7465154Z return func(*args, **kwargs) 2025-10-10T00:59:12.7465451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7465536Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7465786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7465869Z return func(*args, **kwargs) 2025-10-10T00:59:12.7466176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7466283Z self_outputs = self.self( 2025-10-10T00:59:12.7466610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7466700Z return func(*args, **kwargs) 2025-10-10T00:59:12.7467009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7467099Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7467103Z 2025-10-10T00:59:12.7467199Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7467307Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7467429Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7467643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7467714Z res = mod(**inputs) 2025-10-10T00:59:12.7468048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7468135Z outputs = self.bert( 2025-10-10T00:59:12.7468428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7468502Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7468783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7468865Z layer_outputs = layer_module( 2025-10-10T00:59:12.7469085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7469173Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7469411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7469489Z return func(*args, **kwargs) 2025-10-10T00:59:12.7469773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7469856Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7470099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7470168Z return func(*args, **kwargs) 2025-10-10T00:59:12.7470456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7470583Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7470864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7470953Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7470958Z 2025-10-10T00:59:12.7471063Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7471266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7471332Z res = mod(**inputs) 2025-10-10T00:59:12.7471622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7471690Z outputs = self.bert( 2025-10-10T00:59:12.7471971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7472050Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7472332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7472412Z layer_outputs = layer_module( 2025-10-10T00:59:12.7472634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7472731Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7472975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7473043Z return func(*args, **kwargs) 2025-10-10T00:59:12.7473330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7473415Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7473693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7473771Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7474088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7474236Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7474519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7474610Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7474613Z 2025-10-10T00:59:12.7474713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7474908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7474981Z res = mod(**inputs) 2025-10-10T00:59:12.7475267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7475340Z outputs = self.bert( 2025-10-10T00:59:12.7475620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7475701Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7475983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7476054Z layer_outputs = layer_module( 2025-10-10T00:59:12.7476277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7476353Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7476594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7476665Z return func(*args, **kwargs) 2025-10-10T00:59:12.7476957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7477046Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7477300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7477383Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7477684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7477791Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7478065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7478175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7478391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7478462Z return self.act(input) 2025-10-10T00:59:12.7478465Z 2025-10-10T00:59:12.7478570Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7478788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7478852Z res = mod(**inputs) 2025-10-10T00:59:12.7479143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7479210Z outputs = self.bert( 2025-10-10T00:59:12.7479499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7479587Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7479868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7479947Z layer_outputs = layer_module( 2025-10-10T00:59:12.7480169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7480308Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7480551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7480632Z return func(*args, **kwargs) 2025-10-10T00:59:12.7480926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7481016Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7481289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7481371Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7481699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7481836Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7482142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7482237Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7482240Z 2025-10-10T00:59:12.7482345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7482554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7482621Z res = mod(**inputs) 2025-10-10T00:59:12.7482918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7482990Z outputs = self.bert( 2025-10-10T00:59:12.7483275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7483360Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7483648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7483729Z layer_outputs = layer_module( 2025-10-10T00:59:12.7483959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7484046Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7484315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7484396Z return func(*args, **kwargs) 2025-10-10T00:59:12.7484716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7484808Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7485096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7485200Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7485539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7485689Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7485999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7486102Z return input_tensor + hidden_states 2025-10-10T00:59:12.7486105Z 2025-10-10T00:59:12.7486210Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7486417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7486483Z res = mod(**inputs) 2025-10-10T00:59:12.7486794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7486888Z outputs = self.bert( 2025-10-10T00:59:12.7487185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7487267Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7487563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7487639Z layer_outputs = layer_module( 2025-10-10T00:59:12.7487878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7487960Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7488218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7488296Z return func(*args, **kwargs) 2025-10-10T00:59:12.7488596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7488690Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7488938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7489016Z return func(*args, **kwargs) 2025-10-10T00:59:12.7489317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7489399Z self_outputs = self.self( 2025-10-10T00:59:12.7489654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7489722Z return func(*args, **kwargs) 2025-10-10T00:59:12.7490019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7490107Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7490110Z 2025-10-10T00:59:12.7490223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7490428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7490494Z res = mod(**inputs) 2025-10-10T00:59:12.7490811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7490887Z outputs = self.bert( 2025-10-10T00:59:12.7491223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7491305Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7491646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7491747Z layer_outputs = layer_module( 2025-10-10T00:59:12.7491991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7492086Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7492350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7492432Z return func(*args, **kwargs) 2025-10-10T00:59:12.7492774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7492866Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7493129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7493207Z return func(*args, **kwargs) 2025-10-10T00:59:12.7493560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7493638Z self_outputs = self.self( 2025-10-10T00:59:12.7493894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7493976Z return func(*args, **kwargs) 2025-10-10T00:59:12.7494300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7494398Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7494402Z 2025-10-10T00:59:12.7494517Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7494740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7494812Z res = mod(**inputs) 2025-10-10T00:59:12.7495133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7495219Z outputs = self.bert( 2025-10-10T00:59:12.7495530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7495619Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7495943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7496025Z layer_outputs = layer_module( 2025-10-10T00:59:12.7496274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7496361Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7496633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7496715Z return func(*args, **kwargs) 2025-10-10T00:59:12.7497040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7497130Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7497395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7497478Z return func(*args, **kwargs) 2025-10-10T00:59:12.7497804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7497889Z self_outputs = self.self( 2025-10-10T00:59:12.7498162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7498239Z return func(*args, **kwargs) 2025-10-10T00:59:12.7498586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7498679Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7498682Z 2025-10-10T00:59:12.7498782Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7498870Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7498983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7499208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7499298Z res = mod(**inputs) 2025-10-10T00:59:12.7499629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7499704Z outputs = self.bert( 2025-10-10T00:59:12.7500031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7500147Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7500473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7500560Z layer_outputs = layer_module( 2025-10-10T00:59:12.7500805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7500900Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7501179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7501256Z return func(*args, **kwargs) 2025-10-10T00:59:12.7501588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7501678Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7501964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7502041Z return func(*args, **kwargs) 2025-10-10T00:59:12.7502369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7502512Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7502835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7502940Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7502944Z 2025-10-10T00:59:12.7503057Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7503282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7503353Z res = mod(**inputs) 2025-10-10T00:59:12.7503686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7503773Z outputs = self.bert( 2025-10-10T00:59:12.7504095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7504183Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7504506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7504594Z layer_outputs = layer_module( 2025-10-10T00:59:12.7504837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7504925Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7505205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7505305Z return func(*args, **kwargs) 2025-10-10T00:59:12.7505640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7505735Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7506023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7506117Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7506618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7506750Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7507079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7507220Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7507224Z 2025-10-10T00:59:12.7507340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7507569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7507650Z res = mod(**inputs) 2025-10-10T00:59:12.7507973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7508057Z outputs = self.bert( 2025-10-10T00:59:12.7508384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7508465Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7508790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7508872Z layer_outputs = layer_module( 2025-10-10T00:59:12.7509128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7509216Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7509484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7509568Z return func(*args, **kwargs) 2025-10-10T00:59:12.7509884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7509989Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7510279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7510372Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7510736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7510853Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7511179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7511304Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7511551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7511629Z return self.act(input) 2025-10-10T00:59:12.7511634Z 2025-10-10T00:59:12.7511753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7511975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7512045Z res = mod(**inputs) 2025-10-10T00:59:12.7512379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7512475Z outputs = self.bert( 2025-10-10T00:59:12.7512800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7512881Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7513210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7513299Z layer_outputs = layer_module( 2025-10-10T00:59:12.7513563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7513660Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7513925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7514003Z return func(*args, **kwargs) 2025-10-10T00:59:12.7514369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7514461Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7514746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7514829Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7515176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7515320Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7515632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7515731Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7515737Z 2025-10-10T00:59:12.7515849Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7516071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7516144Z res = mod(**inputs) 2025-10-10T00:59:12.7516474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7516555Z outputs = self.bert( 2025-10-10T00:59:12.7516888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7516978Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7517300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7517386Z layer_outputs = layer_module( 2025-10-10T00:59:12.7517635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7517727Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7518004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7518081Z return func(*args, **kwargs) 2025-10-10T00:59:12.7518413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7518509Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7518785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7518867Z return func(*args, **kwargs) 2025-10-10T00:59:12.7519187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7519297Z self_outputs = self.self( 2025-10-10T00:59:12.7519559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7519641Z return func(*args, **kwargs) 2025-10-10T00:59:12.7519955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7520046Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7520050Z 2025-10-10T00:59:12.7520190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7520410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7520489Z res = mod(**inputs) 2025-10-10T00:59:12.7520814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7520890Z outputs = self.bert( 2025-10-10T00:59:12.7521244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7521325Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7521641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7521718Z layer_outputs = layer_module( 2025-10-10T00:59:12.7521964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7522050Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7522310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7522392Z return func(*args, **kwargs) 2025-10-10T00:59:12.7522703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7522802Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7523065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7523139Z return func(*args, **kwargs) 2025-10-10T00:59:12.7523459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7523536Z self_outputs = self.self( 2025-10-10T00:59:12.7523805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7523880Z return func(*args, **kwargs) 2025-10-10T00:59:12.7524196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7524290Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7524297Z 2025-10-10T00:59:12.7524408Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7524630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7524700Z res = mod(**inputs) 2025-10-10T00:59:12.7525020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7525093Z outputs = self.bert( 2025-10-10T00:59:12.7525403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7525488Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7525798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7525883Z layer_outputs = layer_module( 2025-10-10T00:59:12.7526140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7526225Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7526490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7526563Z return func(*args, **kwargs) 2025-10-10T00:59:12.7526877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7526981Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7527249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7527322Z return func(*args, **kwargs) 2025-10-10T00:59:12.7527648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7527751Z self_outputs = self.self( 2025-10-10T00:59:12.7528009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7528088Z return func(*args, **kwargs) 2025-10-10T00:59:12.7528401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7528487Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7528491Z 2025-10-10T00:59:12.7528586Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7528672Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7528790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7529007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7529075Z res = mod(**inputs) 2025-10-10T00:59:12.7529403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7529478Z outputs = self.bert( 2025-10-10T00:59:12.7529803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7529882Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7530210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7530289Z layer_outputs = layer_module( 2025-10-10T00:59:12.7530531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7530624Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7530887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7530972Z return func(*args, **kwargs) 2025-10-10T00:59:12.7531286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7531378Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7531799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7531881Z return func(*args, **kwargs) 2025-10-10T00:59:12.7532209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7532348Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7532663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7532811Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7532819Z 2025-10-10T00:59:12.7532931Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7533158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7533230Z res = mod(**inputs) 2025-10-10T00:59:12.7533561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7533635Z outputs = self.bert( 2025-10-10T00:59:12.7533981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7534069Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7534375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7534458Z layer_outputs = layer_module( 2025-10-10T00:59:12.7534750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7534838Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7535104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7535181Z return func(*args, **kwargs) 2025-10-10T00:59:12.7535498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7535593Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7535877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7535959Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7536299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7536424Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7536732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7536830Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7536834Z 2025-10-10T00:59:12.7536942Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7537159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7537229Z res = mod(**inputs) 2025-10-10T00:59:12.7537540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7537620Z outputs = self.bert( 2025-10-10T00:59:12.7537934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7538024Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7538329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7538407Z layer_outputs = layer_module( 2025-10-10T00:59:12.7538652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7538738Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7539016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7539088Z return func(*args, **kwargs) 2025-10-10T00:59:12.7539377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7539498Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7539771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7539857Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7540168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7540279Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7540576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7540691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7540930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7541005Z return self.act(input) 2025-10-10T00:59:12.7541011Z 2025-10-10T00:59:12.7541166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7541383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7541453Z res = mod(**inputs) 2025-10-10T00:59:12.7541772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7541845Z outputs = self.bert( 2025-10-10T00:59:12.7542163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7542241Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7542554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7542633Z layer_outputs = layer_module( 2025-10-10T00:59:12.7542871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7542967Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7543223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7543306Z return func(*args, **kwargs) 2025-10-10T00:59:12.7543613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7543704Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7543991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7544077Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7544430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7544582Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7544904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7544996Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7545000Z 2025-10-10T00:59:12.7545114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7545342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7545415Z res = mod(**inputs) 2025-10-10T00:59:12.7545743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7545818Z outputs = self.bert( 2025-10-10T00:59:12.7546134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7546243Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7546609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7546705Z layer_outputs = layer_module( 2025-10-10T00:59:12.7546949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7547045Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7547331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7547412Z return func(*args, **kwargs) 2025-10-10T00:59:12.7547724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7547807Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7548104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7548184Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7548496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7548636Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7548921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7549009Z return input_tensor + hidden_states 2025-10-10T00:59:12.7549013Z 2025-10-10T00:59:12.7549116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7549320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7549385Z res = mod(**inputs) 2025-10-10T00:59:12.7549679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7549757Z outputs = self.bert( 2025-10-10T00:59:12.7550042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7550121Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7550405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7550478Z layer_outputs = layer_module( 2025-10-10T00:59:12.7550705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7550788Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7551033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7551105Z return func(*args, **kwargs) 2025-10-10T00:59:12.7551392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7551478Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7551713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7551791Z return func(*args, **kwargs) 2025-10-10T00:59:12.7552077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7552153Z self_outputs = self.self( 2025-10-10T00:59:12.7552391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7552460Z return func(*args, **kwargs) 2025-10-10T00:59:12.7552778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7552863Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7552867Z 2025-10-10T00:59:12.7552979Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7553186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7553258Z res = mod(**inputs) 2025-10-10T00:59:12.7553559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7553629Z outputs = self.bert( 2025-10-10T00:59:12.7553925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7554000Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7554315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7554412Z layer_outputs = layer_module( 2025-10-10T00:59:12.7554634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7554722Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7554965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7555043Z return func(*args, **kwargs) 2025-10-10T00:59:12.7555334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7555419Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7555666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7555741Z return func(*args, **kwargs) 2025-10-10T00:59:12.7556036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7556110Z self_outputs = self.self( 2025-10-10T00:59:12.7556358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7556429Z return func(*args, **kwargs) 2025-10-10T00:59:12.7556720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7556810Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7556813Z 2025-10-10T00:59:12.7556918Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7557122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7557192Z res = mod(**inputs) 2025-10-10T00:59:12.7557486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7557562Z outputs = self.bert( 2025-10-10T00:59:12.7557852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7557932Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7558225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7558305Z layer_outputs = layer_module( 2025-10-10T00:59:12.7558530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7558610Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7558861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7558952Z return func(*args, **kwargs) 2025-10-10T00:59:12.7559248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7559335Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7559578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7559656Z return func(*args, **kwargs) 2025-10-10T00:59:12.7559965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7560045Z self_outputs = self.self( 2025-10-10T00:59:12.7560288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7560359Z return func(*args, **kwargs) 2025-10-10T00:59:12.7560686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7560772Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7560776Z 2025-10-10T00:59:12.7560866Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7560947Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7561059Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7561260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7561325Z res = mod(**inputs) 2025-10-10T00:59:12.7561626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7561695Z outputs = self.bert( 2025-10-10T00:59:12.7561991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7562068Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7562359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7562439Z layer_outputs = layer_module( 2025-10-10T00:59:12.7562661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7562750Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7562993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7563066Z return func(*args, **kwargs) 2025-10-10T00:59:12.7563367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7563454Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7563707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7563777Z return func(*args, **kwargs) 2025-10-10T00:59:12.7564101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7564230Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7564520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7564615Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7564619Z 2025-10-10T00:59:12.7564722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7564940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7565022Z res = mod(**inputs) 2025-10-10T00:59:12.7565312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7565386Z outputs = self.bert( 2025-10-10T00:59:12.7565672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7565753Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7566078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7566160Z layer_outputs = layer_module( 2025-10-10T00:59:12.7566385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7566465Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7566734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7566824Z return func(*args, **kwargs) 2025-10-10T00:59:12.7567129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7567216Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7567483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7567568Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7567908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7568021Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7568310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7568411Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7568414Z 2025-10-10T00:59:12.7568517Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7568714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7568785Z res = mod(**inputs) 2025-10-10T00:59:12.7569076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7569149Z outputs = self.bert( 2025-10-10T00:59:12.7569437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7569509Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7569805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7569881Z layer_outputs = layer_module( 2025-10-10T00:59:12.7570111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7570189Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7570437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7570506Z return func(*args, **kwargs) 2025-10-10T00:59:12.7570796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7570886Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7571148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7571231Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7571561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7571665Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7571952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7572063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7572302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7572375Z return self.act(input) 2025-10-10T00:59:12.7572379Z 2025-10-10T00:59:12.7572486Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7572680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7572744Z res = mod(**inputs) 2025-10-10T00:59:12.7573077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7573162Z outputs = self.bert( 2025-10-10T00:59:12.7573451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7573523Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7573808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7573889Z layer_outputs = layer_module( 2025-10-10T00:59:12.7574108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7574193Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7574430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7574508Z return func(*args, **kwargs) 2025-10-10T00:59:12.7574793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7574875Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7575140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7575215Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7575530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7575660Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7575942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7576037Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7576040Z 2025-10-10T00:59:12.7576142Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7576346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7576411Z res = mod(**inputs) 2025-10-10T00:59:12.7576705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7576774Z outputs = self.bert( 2025-10-10T00:59:12.7577059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7577141Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7577428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7577525Z layer_outputs = layer_module( 2025-10-10T00:59:12.7577747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7577826Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7578069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7578140Z return func(*args, **kwargs) 2025-10-10T00:59:12.7578428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7578528Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7578776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7578845Z return func(*args, **kwargs) 2025-10-10T00:59:12.7579154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7579258Z self_outputs = self.self( 2025-10-10T00:59:12.7579494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7579568Z return func(*args, **kwargs) 2025-10-10T00:59:12.7579890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7579973Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7579976Z 2025-10-10T00:59:12.7580089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7580285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7580358Z res = mod(**inputs) 2025-10-10T00:59:12.7580642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7580717Z outputs = self.bert( 2025-10-10T00:59:12.7581013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7581087Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7581385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7581458Z layer_outputs = layer_module( 2025-10-10T00:59:12.7581693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7581774Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7582016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7582095Z return func(*args, **kwargs) 2025-10-10T00:59:12.7582387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7582480Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7582721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7582791Z return func(*args, **kwargs) 2025-10-10T00:59:12.7583089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7583167Z self_outputs = self.self( 2025-10-10T00:59:12.7583433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7583505Z return func(*args, **kwargs) 2025-10-10T00:59:12.7583817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7583924Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7583928Z 2025-10-10T00:59:12.7584040Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7584259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7584328Z res = mod(**inputs) 2025-10-10T00:59:12.7584644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7584717Z outputs = self.bert( 2025-10-10T00:59:12.7585038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7585126Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7585433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7585558Z layer_outputs = layer_module( 2025-10-10T00:59:12.7585797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7585890Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7586148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7586225Z return func(*args, **kwargs) 2025-10-10T00:59:12.7586625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7586724Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7586996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7587072Z return func(*args, **kwargs) 2025-10-10T00:59:12.7587389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7587497Z self_outputs = self.self( 2025-10-10T00:59:12.7587765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7587849Z return func(*args, **kwargs) 2025-10-10T00:59:12.7588164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7588244Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7588256Z 2025-10-10T00:59:12.7588335Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7588414Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7588523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7588713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7588788Z res = mod(**inputs) 2025-10-10T00:59:12.7589069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7589134Z outputs = self.bert( 2025-10-10T00:59:12.7589416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7589488Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7589771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7589842Z layer_outputs = layer_module( 2025-10-10T00:59:12.7590063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7590147Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7590384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7590477Z return func(*args, **kwargs) 2025-10-10T00:59:12.7590758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7590838Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7591079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7591148Z return func(*args, **kwargs) 2025-10-10T00:59:12.7592095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7592234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7592525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7592644Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7592649Z 2025-10-10T00:59:12.7592752Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7592954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7593020Z res = mod(**inputs) 2025-10-10T00:59:12.7593317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7593385Z outputs = self.bert( 2025-10-10T00:59:12.7593676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7593757Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7594047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7594133Z layer_outputs = layer_module( 2025-10-10T00:59:12.7594365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7594451Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7594687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7594756Z return func(*args, **kwargs) 2025-10-10T00:59:12.7595049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7595131Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7595386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7595463Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7595784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7595902Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7596189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7596286Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7596290Z 2025-10-10T00:59:12.7596394Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7596603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7596670Z res = mod(**inputs) 2025-10-10T00:59:12.7596964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7597042Z outputs = self.bert( 2025-10-10T00:59:12.7597360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7597443Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7597725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7597796Z layer_outputs = layer_module( 2025-10-10T00:59:12.7598022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7598116Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7598362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7598431Z return func(*args, **kwargs) 2025-10-10T00:59:12.7598730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7598852Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7599120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7599208Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7599527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7599641Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7599934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7600050Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7600276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7600352Z return self.act(input) 2025-10-10T00:59:12.7600359Z 2025-10-10T00:59:12.7600470Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7600671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7600746Z res = mod(**inputs) 2025-10-10T00:59:12.7601042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7601112Z outputs = self.bert( 2025-10-10T00:59:12.7601414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7601488Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7601785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7601859Z layer_outputs = layer_module( 2025-10-10T00:59:12.7602090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7602179Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7602424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7602502Z return func(*args, **kwargs) 2025-10-10T00:59:12.7602795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7602887Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7603151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7603230Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7603558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7603714Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7604016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7604103Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7604107Z 2025-10-10T00:59:12.7604210Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7604460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7604528Z res = mod(**inputs) 2025-10-10T00:59:12.7604829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7604899Z outputs = self.bert( 2025-10-10T00:59:12.7605212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7605304Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7605591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7605671Z layer_outputs = layer_module( 2025-10-10T00:59:12.7605896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7605984Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7606228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7606299Z return func(*args, **kwargs) 2025-10-10T00:59:12.7606596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7606682Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7606954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7607033Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7607355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7607492Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7607784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7607873Z return input_tensor + hidden_states 2025-10-10T00:59:12.7607876Z 2025-10-10T00:59:12.7607980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7608188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7608261Z res = mod(**inputs) 2025-10-10T00:59:12.7608553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7608629Z outputs = self.bert( 2025-10-10T00:59:12.7608918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7608996Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7609286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7609367Z layer_outputs = layer_module( 2025-10-10T00:59:12.7609592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7609673Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7609970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7610044Z return func(*args, **kwargs) 2025-10-10T00:59:12.7610359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7610449Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7610708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7610791Z return func(*args, **kwargs) 2025-10-10T00:59:12.7611116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7611201Z self_outputs = self.self( 2025-10-10T00:59:12.7611461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7611565Z return func(*args, **kwargs) 2025-10-10T00:59:12.7611862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7611947Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7611950Z 2025-10-10T00:59:12.7612061Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7612262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7612335Z res = mod(**inputs) 2025-10-10T00:59:12.7612630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7612700Z outputs = self.bert( 2025-10-10T00:59:12.7612996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7613077Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7613392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7613471Z layer_outputs = layer_module( 2025-10-10T00:59:12.7613708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7613799Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7614057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7614139Z return func(*args, **kwargs) 2025-10-10T00:59:12.7614445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7614543Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7614800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7614878Z return func(*args, **kwargs) 2025-10-10T00:59:12.7615192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7615268Z self_outputs = self.self( 2025-10-10T00:59:12.7615529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7615604Z return func(*args, **kwargs) 2025-10-10T00:59:12.7615909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7616001Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7616005Z 2025-10-10T00:59:12.7616115Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7616335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7616428Z res = mod(**inputs) 2025-10-10T00:59:12.7616744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7616818Z outputs = self.bert( 2025-10-10T00:59:12.7617126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7617212Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7617533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7617619Z layer_outputs = layer_module( 2025-10-10T00:59:12.7617858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7617941Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7618240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7618316Z return func(*args, **kwargs) 2025-10-10T00:59:12.7618629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7618717Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7618974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7619058Z return func(*args, **kwargs) 2025-10-10T00:59:12.7619369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7619454Z self_outputs = self.self( 2025-10-10T00:59:12.7619712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7619798Z return func(*args, **kwargs) 2025-10-10T00:59:12.7620105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7620193Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7620197Z 2025-10-10T00:59:12.7620292Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7620379Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7620497Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7620710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7620780Z res = mod(**inputs) 2025-10-10T00:59:12.7621100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7621172Z outputs = self.bert( 2025-10-10T00:59:12.7621487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7621566Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7621873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7621958Z layer_outputs = layer_module( 2025-10-10T00:59:12.7622198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7622289Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7622547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7622628Z return func(*args, **kwargs) 2025-10-10T00:59:12.7622941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7623052Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7623315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7623389Z return func(*args, **kwargs) 2025-10-10T00:59:12.7623702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7623838Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7624162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7624263Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7624266Z 2025-10-10T00:59:12.7624376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7624615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7624703Z res = mod(**inputs) 2025-10-10T00:59:12.7625023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7625094Z outputs = self.bert( 2025-10-10T00:59:12.7625401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7625487Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7625794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7625877Z layer_outputs = layer_module( 2025-10-10T00:59:12.7626116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7626199Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7626555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7626638Z return func(*args, **kwargs) 2025-10-10T00:59:12.7626962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7627057Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7627377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7627467Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7627824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7627945Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7628252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7628352Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7628356Z 2025-10-10T00:59:12.7628466Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7628681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7628760Z res = mod(**inputs) 2025-10-10T00:59:12.7629071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7629154Z outputs = self.bert( 2025-10-10T00:59:12.7629459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7629549Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7629856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7629966Z layer_outputs = layer_module( 2025-10-10T00:59:12.7630210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7630297Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7630561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7630637Z return func(*args, **kwargs) 2025-10-10T00:59:12.7630959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7631060Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7631336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7631428Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7631979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7632103Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7632416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7632544Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7632793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7632884Z return self.act(input) 2025-10-10T00:59:12.7632888Z 2025-10-10T00:59:12.7633009Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7633225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7633297Z res = mod(**inputs) 2025-10-10T00:59:12.7633620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7633694Z outputs = self.bert( 2025-10-10T00:59:12.7634016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7634095Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7634412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7634490Z layer_outputs = layer_module( 2025-10-10T00:59:12.7634728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7634824Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7635085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7635171Z return func(*args, **kwargs) 2025-10-10T00:59:12.7635478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7635570Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7635857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7635940Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7636291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7636436Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7636750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7636872Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7636876Z 2025-10-10T00:59:12.7636987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7637208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7637279Z res = mod(**inputs) 2025-10-10T00:59:12.7637603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7637699Z outputs = self.bert( 2025-10-10T00:59:12.7638019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7638107Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7638421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7638550Z layer_outputs = layer_module( 2025-10-10T00:59:12.7638791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7638883Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7639144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7639219Z return func(*args, **kwargs) 2025-10-10T00:59:12.7639543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7639632Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7639903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7639972Z return func(*args, **kwargs) 2025-10-10T00:59:12.7640268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7640349Z self_outputs = self.self( 2025-10-10T00:59:12.7640596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7640671Z return func(*args, **kwargs) 2025-10-10T00:59:12.7640966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7641056Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7641065Z 2025-10-10T00:59:12.7641180Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7641394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7641472Z res = mod(**inputs) 2025-10-10T00:59:12.7641794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7641877Z outputs = self.bert( 2025-10-10T00:59:12.7642184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7642261Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7642577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7642655Z layer_outputs = layer_module( 2025-10-10T00:59:12.7642901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7642985Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7643247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7643358Z return func(*args, **kwargs) 2025-10-10T00:59:12.7643651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7643743Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7643987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7644064Z return func(*args, **kwargs) 2025-10-10T00:59:12.7644372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7644447Z self_outputs = self.self( 2025-10-10T00:59:12.7644696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7644766Z return func(*args, **kwargs) 2025-10-10T00:59:12.7645082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7645180Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7645184Z 2025-10-10T00:59:12.7645290Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7645498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7645567Z res = mod(**inputs) 2025-10-10T00:59:12.7645866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7645935Z outputs = self.bert( 2025-10-10T00:59:12.7646231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7646305Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7646596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7646683Z layer_outputs = layer_module( 2025-10-10T00:59:12.7646909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7646999Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7647242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7647311Z return func(*args, **kwargs) 2025-10-10T00:59:12.7647610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7647694Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7647944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7648016Z return func(*args, **kwargs) 2025-10-10T00:59:12.7648307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7648386Z self_outputs = self.self( 2025-10-10T00:59:12.7648629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7648706Z return func(*args, **kwargs) 2025-10-10T00:59:12.7648994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7649084Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7649087Z 2025-10-10T00:59:12.7649172Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7649252Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7649363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7649563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7649658Z res = mod(**inputs) 2025-10-10T00:59:12.7649951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7650018Z outputs = self.bert( 2025-10-10T00:59:12.7650316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7650391Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7650702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7650778Z layer_outputs = layer_module( 2025-10-10T00:59:12.7651004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7651093Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7651384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7651465Z return func(*args, **kwargs) 2025-10-10T00:59:12.7651757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7651849Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7652095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7652170Z return func(*args, **kwargs) 2025-10-10T00:59:12.7652490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7652628Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7652948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7653042Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7653046Z 2025-10-10T00:59:12.7653166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7653381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7653452Z res = mod(**inputs) 2025-10-10T00:59:12.7653771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7653846Z outputs = self.bert( 2025-10-10T00:59:12.7654165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7654245Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7654557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7654644Z layer_outputs = layer_module( 2025-10-10T00:59:12.7654882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7654974Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7655235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7655310Z return func(*args, **kwargs) 2025-10-10T00:59:12.7655628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7655720Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7656009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7656093Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7656456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7656568Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7656876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7656975Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7656979Z 2025-10-10T00:59:12.7657104Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7657326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7657396Z res = mod(**inputs) 2025-10-10T00:59:12.7657719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7657802Z outputs = self.bert( 2025-10-10T00:59:12.7658138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7658225Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7658537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7658621Z layer_outputs = layer_module( 2025-10-10T00:59:12.7658861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7658947Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7659216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7659289Z return func(*args, **kwargs) 2025-10-10T00:59:12.7659607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7659701Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7659981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7660069Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7660412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7660532Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7660843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7660973Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7661207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7661288Z return self.act(input) 2025-10-10T00:59:12.7661292Z 2025-10-10T00:59:12.7661410Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7661624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7661702Z res = mod(**inputs) 2025-10-10T00:59:12.7662022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7662095Z outputs = self.bert( 2025-10-10T00:59:12.7662423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7662503Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7662826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7662932Z layer_outputs = layer_module( 2025-10-10T00:59:12.7663186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7663278Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7663547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7663632Z return func(*args, **kwargs) 2025-10-10T00:59:12.7663977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7664079Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7664363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7664449Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7664825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7664993Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7665323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7665415Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7665419Z 2025-10-10T00:59:12.7665541Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7665763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7665836Z res = mod(**inputs) 2025-10-10T00:59:12.7666166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7666242Z outputs = self.bert( 2025-10-10T00:59:12.7666642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7666731Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7667057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7667149Z layer_outputs = layer_module( 2025-10-10T00:59:12.7667404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7667507Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7667794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7667881Z return func(*args, **kwargs) 2025-10-10T00:59:12.7668214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7668310Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7668598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7668680Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7669035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7669177Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7669493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7669586Z return input_tensor + hidden_states 2025-10-10T00:59:12.7669590Z 2025-10-10T00:59:12.7669701Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7669924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7670020Z res = mod(**inputs) 2025-10-10T00:59:12.7670347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7670420Z outputs = self.bert( 2025-10-10T00:59:12.7670733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7670821Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7671151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7671239Z layer_outputs = layer_module( 2025-10-10T00:59:12.7671478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7671565Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7671884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7671960Z return func(*args, **kwargs) 2025-10-10T00:59:12.7672284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7672375Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7672653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7672729Z return func(*args, **kwargs) 2025-10-10T00:59:12.7673039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7673123Z self_outputs = self.self( 2025-10-10T00:59:12.7673393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7673478Z return func(*args, **kwargs) 2025-10-10T00:59:12.7673793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7673882Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7673886Z 2025-10-10T00:59:12.7674004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7674219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7674297Z res = mod(**inputs) 2025-10-10T00:59:12.7674627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7674701Z outputs = self.bert( 2025-10-10T00:59:12.7675042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7675124Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7675445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7675524Z layer_outputs = layer_module( 2025-10-10T00:59:12.7675770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7675854Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7676125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7676207Z return func(*args, **kwargs) 2025-10-10T00:59:12.7676523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7676619Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7676914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7676988Z return func(*args, **kwargs) 2025-10-10T00:59:12.7677310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7677386Z self_outputs = self.self( 2025-10-10T00:59:12.7677649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7677740Z return func(*args, **kwargs) 2025-10-10T00:59:12.7678055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7678140Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7678144Z 2025-10-10T00:59:12.7678254Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7678520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7678593Z res = mod(**inputs) 2025-10-10T00:59:12.7678918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7678990Z outputs = self.bert( 2025-10-10T00:59:12.7679303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7679391Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7679710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7679795Z layer_outputs = layer_module( 2025-10-10T00:59:12.7680032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7680123Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7680387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7680461Z return func(*args, **kwargs) 2025-10-10T00:59:12.7680777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7680866Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7681129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7681203Z return func(*args, **kwargs) 2025-10-10T00:59:12.7681510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7681595Z self_outputs = self.self( 2025-10-10T00:59:12.7681852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7681937Z return func(*args, **kwargs) 2025-10-10T00:59:12.7682243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7682331Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7682342Z 2025-10-10T00:59:12.7682428Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7682514Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7682635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7682845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7682915Z res = mod(**inputs) 2025-10-10T00:59:12.7683237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7683333Z outputs = self.bert( 2025-10-10T00:59:12.7683654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7683733Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7684058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7684136Z layer_outputs = layer_module( 2025-10-10T00:59:12.7684398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7684494Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7684750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7684833Z return func(*args, **kwargs) 2025-10-10T00:59:12.7685162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7685268Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7685531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7685606Z return func(*args, **kwargs) 2025-10-10T00:59:12.7685923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7686063Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7686376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7686468Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7686472Z 2025-10-10T00:59:12.7686582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7686808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7686878Z res = mod(**inputs) 2025-10-10T00:59:12.7687201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7687273Z outputs = self.bert( 2025-10-10T00:59:12.7687578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7687666Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7687972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7688057Z layer_outputs = layer_module( 2025-10-10T00:59:12.7688294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7688390Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7688646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7688721Z return func(*args, **kwargs) 2025-10-10T00:59:12.7689034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7689125Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7689409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7689493Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7689835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7689954Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7690280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7690378Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7690381Z 2025-10-10T00:59:12.7690491Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7690707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7690775Z res = mod(**inputs) 2025-10-10T00:59:12.7691105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7691188Z outputs = self.bert( 2025-10-10T00:59:12.7691495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7691583Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7691914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7692011Z layer_outputs = layer_module( 2025-10-10T00:59:12.7692256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7692342Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7692605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7692681Z return func(*args, **kwargs) 2025-10-10T00:59:12.7692993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7693083Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7693360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7693456Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7693796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7693915Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7694222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7694343Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7694582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7694657Z return self.act(input) 2025-10-10T00:59:12.7694661Z 2025-10-10T00:59:12.7694780Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7694992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7695075Z res = mod(**inputs) 2025-10-10T00:59:12.7695384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7695456Z outputs = self.bert( 2025-10-10T00:59:12.7695766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7695846Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7696161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7696240Z layer_outputs = layer_module( 2025-10-10T00:59:12.7696479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7696574Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7696858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7696944Z return func(*args, **kwargs) 2025-10-10T00:59:12.7697251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7697349Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7697631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7697732Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7698089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7698235Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7698581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7698691Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7698695Z 2025-10-10T00:59:12.7698805Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7699022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7699091Z res = mod(**inputs) 2025-10-10T00:59:12.7699422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7699498Z outputs = self.bert( 2025-10-10T00:59:12.7699829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7699910Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7700243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7700334Z layer_outputs = layer_module( 2025-10-10T00:59:12.7700586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7700681Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7700955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7701033Z return func(*args, **kwargs) 2025-10-10T00:59:12.7701377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7701472Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7701749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7701827Z return func(*args, **kwargs) 2025-10-10T00:59:12.7702156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7702246Z self_outputs = self.self( 2025-10-10T00:59:12.7702515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7702599Z return func(*args, **kwargs) 2025-10-10T00:59:12.7702927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7703027Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7703031Z 2025-10-10T00:59:12.7703146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7703370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7703452Z res = mod(**inputs) 2025-10-10T00:59:12.7703792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7703875Z outputs = self.bert( 2025-10-10T00:59:12.7704204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7704285Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7704661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7704769Z layer_outputs = layer_module( 2025-10-10T00:59:12.7705025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7705116Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7705393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7705508Z return func(*args, **kwargs) 2025-10-10T00:59:12.7705823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7705923Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7706185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7706271Z return func(*args, **kwargs) 2025-10-10T00:59:12.7706665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7706750Z self_outputs = self.self( 2025-10-10T00:59:12.7707024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7707101Z return func(*args, **kwargs) 2025-10-10T00:59:12.7707430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7707520Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7707523Z 2025-10-10T00:59:12.7707647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7707867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7707939Z res = mod(**inputs) 2025-10-10T00:59:12.7708270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7708345Z outputs = self.bert( 2025-10-10T00:59:12.7708669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7708751Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7709070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7709158Z layer_outputs = layer_module( 2025-10-10T00:59:12.7709404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7709502Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7709769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7709845Z return func(*args, **kwargs) 2025-10-10T00:59:12.7710175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7710269Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7710540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7710645Z return func(*args, **kwargs) 2025-10-10T00:59:12.7710970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7711049Z self_outputs = self.self( 2025-10-10T00:59:12.7711315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7711399Z return func(*args, **kwargs) 2025-10-10T00:59:12.7711738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7711840Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7711843Z 2025-10-10T00:59:12.7711932Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7712021Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7712146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7712430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7712512Z res = mod(**inputs) 2025-10-10T00:59:12.7712833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7712907Z outputs = self.bert( 2025-10-10T00:59:12.7713234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7713315Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7713639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7713720Z layer_outputs = layer_module( 2025-10-10T00:59:12.7713975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7714067Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7714333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7714418Z return func(*args, **kwargs) 2025-10-10T00:59:12.7714735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7714835Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7715102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7715179Z return func(*args, **kwargs) 2025-10-10T00:59:12.7715505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7715647Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7715980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7716077Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7716081Z 2025-10-10T00:59:12.7716205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7716423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7716494Z res = mod(**inputs) 2025-10-10T00:59:12.7716823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7716898Z outputs = self.bert( 2025-10-10T00:59:12.7717220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7717301Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7717637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7717728Z layer_outputs = layer_module( 2025-10-10T00:59:12.7717977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7718076Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7718340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7718423Z return func(*args, **kwargs) 2025-10-10T00:59:12.7718762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7718858Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7719151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7719276Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7719636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7719755Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7720075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7720175Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7720179Z 2025-10-10T00:59:12.7720294Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7720524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7720597Z res = mod(**inputs) 2025-10-10T00:59:12.7720935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7721015Z outputs = self.bert( 2025-10-10T00:59:12.7721334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7721422Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7721744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7721834Z layer_outputs = layer_module( 2025-10-10T00:59:12.7722085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7722173Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7722448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7722524Z return func(*args, **kwargs) 2025-10-10T00:59:12.7722853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7722946Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7723244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7723331Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7723683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7723809Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7724127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7724263Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7724524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7724605Z return self.act(input) 2025-10-10T00:59:12.7724609Z 2025-10-10T00:59:12.7724729Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7724945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7725023Z res = mod(**inputs) 2025-10-10T00:59:12.7725351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7725450Z outputs = self.bert( 2025-10-10T00:59:12.7725767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7725848Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7726188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7726287Z layer_outputs = layer_module( 2025-10-10T00:59:12.7726538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7726628Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7726894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7726979Z return func(*args, **kwargs) 2025-10-10T00:59:12.7727306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7727406Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7727696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7727781Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7728145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7728294Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7728617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7728707Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7728711Z 2025-10-10T00:59:12.7728833Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7729053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7729125Z res = mod(**inputs) 2025-10-10T00:59:12.7729461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7729541Z outputs = self.bert( 2025-10-10T00:59:12.7729873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7729953Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7730284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7730365Z layer_outputs = layer_module( 2025-10-10T00:59:12.7730617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7730713Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7730987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7731073Z return func(*args, **kwargs) 2025-10-10T00:59:12.7731397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7731671Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7731976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7732062Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7732423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7732615Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7733007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7733095Z return input_tensor + hidden_states 2025-10-10T00:59:12.7733099Z 2025-10-10T00:59:12.7733215Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7733495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7733570Z res = mod(**inputs) 2025-10-10T00:59:12.7733895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7733972Z outputs = self.bert( 2025-10-10T00:59:12.7734297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7734391Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7734720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7734809Z layer_outputs = layer_module( 2025-10-10T00:59:12.7735065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7735161Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7735440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7735513Z return func(*args, **kwargs) 2025-10-10T00:59:12.7735811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7735898Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7736149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7736220Z return func(*args, **kwargs) 2025-10-10T00:59:12.7736511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7736593Z self_outputs = self.self( 2025-10-10T00:59:12.7736838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7736919Z return func(*args, **kwargs) 2025-10-10T00:59:12.7737212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7737296Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7737307Z 2025-10-10T00:59:12.7737412Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7737613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7737686Z res = mod(**inputs) 2025-10-10T00:59:12.7737977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7738054Z outputs = self.bert( 2025-10-10T00:59:12.7738343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7738444Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7738739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7738814Z layer_outputs = layer_module( 2025-10-10T00:59:12.7739049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7739131Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7739391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7739471Z return func(*args, **kwargs) 2025-10-10T00:59:12.7739758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7739885Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7740130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7740208Z return func(*args, **kwargs) 2025-10-10T00:59:12.7740503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7740575Z self_outputs = self.self( 2025-10-10T00:59:12.7740828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7740898Z return func(*args, **kwargs) 2025-10-10T00:59:12.7741194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7741274Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7741278Z 2025-10-10T00:59:12.7741385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7741610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7741681Z res = mod(**inputs) 2025-10-10T00:59:12.7742002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7742074Z outputs = self.bert( 2025-10-10T00:59:12.7742382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7742468Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7742778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7742865Z layer_outputs = layer_module( 2025-10-10T00:59:12.7743107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7743201Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7743461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7743536Z return func(*args, **kwargs) 2025-10-10T00:59:12.7743853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7743944Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7744212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7744286Z return func(*args, **kwargs) 2025-10-10T00:59:12.7744595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7744677Z self_outputs = self.self( 2025-10-10T00:59:12.7744957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7745039Z return func(*args, **kwargs) 2025-10-10T00:59:12.7745347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7745440Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7745444Z 2025-10-10T00:59:12.7745532Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7745618Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7745752Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7745965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7746044Z res = mod(**inputs) 2025-10-10T00:59:12.7746484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7746631Z outputs = self.bert( 2025-10-10T00:59:12.7746964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7747048Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7747385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7747463Z layer_outputs = layer_module( 2025-10-10T00:59:12.7747709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7747803Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7748062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7748157Z return func(*args, **kwargs) 2025-10-10T00:59:12.7748462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7748558Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7748809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7748882Z return func(*args, **kwargs) 2025-10-10T00:59:12.7749190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7749328Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7749644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7749734Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7749738Z 2025-10-10T00:59:12.7749853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7750074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7750144Z res = mod(**inputs) 2025-10-10T00:59:12.7750475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7750545Z outputs = self.bert( 2025-10-10T00:59:12.7750852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7750930Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7751228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7751311Z layer_outputs = layer_module( 2025-10-10T00:59:12.7751545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7751660Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7751908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7751981Z return func(*args, **kwargs) 2025-10-10T00:59:12.7752285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7752375Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7752670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7752754Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7753103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7753231Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7753547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7753641Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7753645Z 2025-10-10T00:59:12.7753751Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7753957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7754024Z res = mod(**inputs) 2025-10-10T00:59:12.7754323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7754399Z outputs = self.bert( 2025-10-10T00:59:12.7754694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7754776Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7755073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7755153Z layer_outputs = layer_module( 2025-10-10T00:59:12.7755382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7755464Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7755719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7755791Z return func(*args, **kwargs) 2025-10-10T00:59:12.7756091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7756174Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7756440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7756530Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7756856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7756969Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7757260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7757384Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7757606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7757677Z return self.act(input) 2025-10-10T00:59:12.7757681Z 2025-10-10T00:59:12.7757790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7758012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7758086Z res = mod(**inputs) 2025-10-10T00:59:12.7758379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7758447Z outputs = self.bert( 2025-10-10T00:59:12.7758744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7758816Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7759130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7759205Z layer_outputs = layer_module( 2025-10-10T00:59:12.7759430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7759538Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7759796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7759874Z return func(*args, **kwargs) 2025-10-10T00:59:12.7760160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7760251Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7760512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7760589Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7760911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7761044Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7761344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7761428Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7761431Z 2025-10-10T00:59:12.7761544Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7761745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7761810Z res = mod(**inputs) 2025-10-10T00:59:12.7762111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7762179Z outputs = self.bert( 2025-10-10T00:59:12.7762477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7762551Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7762845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7762925Z layer_outputs = layer_module( 2025-10-10T00:59:12.7763149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7763237Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7763479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7763552Z return func(*args, **kwargs) 2025-10-10T00:59:12.7763857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7763940Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7764183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7764273Z return func(*args, **kwargs) 2025-10-10T00:59:12.7764564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7764636Z self_outputs = self.self( 2025-10-10T00:59:12.7764874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7764950Z return func(*args, **kwargs) 2025-10-10T00:59:12.7765254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-10-10T00:59:12.7765344Z query_layer = self.query(hidden_states) 2025-10-10T00:59:12.7765348Z 2025-10-10T00:59:12.7765449Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7765642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7765749Z res = mod(**inputs) 2025-10-10T00:59:12.7766033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7766106Z outputs = self.bert( 2025-10-10T00:59:12.7766389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7766469Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7766760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7766832Z layer_outputs = layer_module( 2025-10-10T00:59:12.7767059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7767137Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7767381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7767453Z return func(*args, **kwargs) 2025-10-10T00:59:12.7767734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7767824Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7768058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7768133Z return func(*args, **kwargs) 2025-10-10T00:59:12.7768418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7768495Z self_outputs = self.self( 2025-10-10T00:59:12.7768731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7768801Z return func(*args, **kwargs) 2025-10-10T00:59:12.7769091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-10-10T00:59:12.7769172Z key_layer = self.key(current_states) 2025-10-10T00:59:12.7769175Z 2025-10-10T00:59:12.7769284Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7769477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7769542Z res = mod(**inputs) 2025-10-10T00:59:12.7769836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7769903Z outputs = self.bert( 2025-10-10T00:59:12.7770189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7770262Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7770559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7770638Z layer_outputs = layer_module( 2025-10-10T00:59:12.7770854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7770941Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7771177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7771267Z return func(*args, **kwargs) 2025-10-10T00:59:12.7771555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7771637Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7771893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7771981Z return func(*args, **kwargs) 2025-10-10T00:59:12.7772277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-10-10T00:59:12.7772348Z self_outputs = self.self( 2025-10-10T00:59:12.7772590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7772665Z return func(*args, **kwargs) 2025-10-10T00:59:12.7772954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-10-10T00:59:12.7773043Z value_layer = self.value(current_states) 2025-10-10T00:59:12.7773046Z 2025-10-10T00:59:12.7773128Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7773216Z cudagraph partition due to non gpu ops 2025-10-10T00:59:12.7773321Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7773526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7773599Z res = mod(**inputs) 2025-10-10T00:59:12.7773898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7773973Z outputs = self.bert( 2025-10-10T00:59:12.7774275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7774349Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7774644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7774716Z layer_outputs = layer_module( 2025-10-10T00:59:12.7774944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7775025Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7775268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7775344Z return func(*args, **kwargs) 2025-10-10T00:59:12.7775633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-10-10T00:59:12.7775722Z self_attention_outputs = self.attention( 2025-10-10T00:59:12.7775964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7776039Z return func(*args, **kwargs) 2025-10-10T00:59:12.7776328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-10-10T00:59:12.7776453Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T00:59:12.7776759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-10-10T00:59:12.7776843Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7776846Z 2025-10-10T00:59:12.7776954Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7777148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7777214Z res = mod(**inputs) 2025-10-10T00:59:12.7777523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7777593Z outputs = self.bert( 2025-10-10T00:59:12.7777885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7777957Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7778286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7778359Z layer_outputs = layer_module( 2025-10-10T00:59:12.7778576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7778664Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7778900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7778977Z return func(*args, **kwargs) 2025-10-10T00:59:12.7779261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7779345Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7779611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7779690Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7780009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7780115Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7780413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-10-10T00:59:12.7780499Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7780503Z 2025-10-10T00:59:12.7780606Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7780812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7780879Z res = mod(**inputs) 2025-10-10T00:59:12.7781180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7781252Z outputs = self.bert( 2025-10-10T00:59:12.7781539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7781622Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7781909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7781990Z layer_outputs = layer_module( 2025-10-10T00:59:12.7782213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7782301Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7782544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7782635Z return func(*args, **kwargs) 2025-10-10T00:59:12.7782935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7783020Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7783289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7783367Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7783706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-10-10T00:59:12.7783821Z intermediate_output = self.intermediate(ln_output) 2025-10-10T00:59:12.7784113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-10-10T00:59:12.7784236Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T00:59:12.7784490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T00:59:12.7784574Z return self.act(input) 2025-10-10T00:59:12.7784577Z 2025-10-10T00:59:12.7784685Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7784888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7784966Z res = mod(**inputs) 2025-10-10T00:59:12.7785262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7785343Z outputs = self.bert( 2025-10-10T00:59:12.7785641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7785724Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7786056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7786140Z layer_outputs = layer_module( 2025-10-10T00:59:12.7786454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7786550Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7786818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7786905Z return func(*args, **kwargs) 2025-10-10T00:59:12.7787238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7787343Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7787637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7787732Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7788081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7788214Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7788519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-10-10T00:59:12.7788605Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7788609Z 2025-10-10T00:59:12.7788723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7788925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7789000Z res = mod(**inputs) 2025-10-10T00:59:12.7789297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-10-10T00:59:12.7789392Z outputs = self.bert( 2025-10-10T00:59:12.7789684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-10-10T00:59:12.7789757Z encoder_outputs = self.encoder( 2025-10-10T00:59:12.7790053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-10-10T00:59:12.7790125Z layer_outputs = layer_module( 2025-10-10T00:59:12.7790365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T00:59:12.7790454Z return super().__call__(*args, **kwargs) 2025-10-10T00:59:12.7790700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T00:59:12.7790779Z return func(*args, **kwargs) 2025-10-10T00:59:12.7791103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-10-10T00:59:12.7791188Z layer_output = apply_chunking_to_forward( 2025-10-10T00:59:12.7791457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T00:59:12.7791540Z return forward_fn(*input_tensors) 2025-10-10T00:59:12.7791870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-10-10T00:59:12.7792005Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T00:59:12.7828583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-10-10T00:59:12.7828812Z return input_tensor + hidden_states 2025-10-10T00:59:12.7828844Z 2025-10-10T00:59:12.7829007Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7829255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7829378Z res = mod(**inputs) 2025-10-10T00:59:12.7829802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-10-10T00:59:12.7829958Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:59:12.7830376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-10-10T00:59:12.7830516Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:59:12.7830837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 632, in forward 2025-10-10T00:59:12.7830942Z hidden_states = self.transform(hidden_states) 2025-10-10T00:59:12.7831274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 607, in forward 2025-10-10T00:59:12.7831370Z hidden_states = self.dense(hidden_states) 2025-10-10T00:59:12.7831375Z 2025-10-10T00:59:12.7831682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7831920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7831998Z res = mod(**inputs) 2025-10-10T00:59:12.7832352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-10-10T00:59:12.7832454Z prediction_scores = self.cls(sequence_output) 2025-10-10T00:59:12.7832790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-10-10T00:59:12.7832911Z prediction_scores = self.predictions(sequence_output) 2025-10-10T00:59:12.7833374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 633, in forward 2025-10-10T00:59:12.7833475Z hidden_states = self.decoder(hidden_states) 2025-10-10T00:59:12.7833479Z 2025-10-10T00:59:12.7833594Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T00:59:12.7833822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T00:59:12.7833894Z res = mod(**inputs) 2025-10-10T00:59:12.7834271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1078, in forward 2025-10-10T00:59:12.7834359Z lm_loss = self.loss_function( 2025-10-10T00:59:12.7834638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T00:59:12.7834882Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T00:59:12.7835207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T00:59:12.7835429Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T00:59:12.7835434Z 2025-10-10T00:59:24.8490516Z Compilation time (from dynamo_timed): 25.907739453 2025-10-10T00:59:24.8530747Z pass 2025-10-10T00:59:24.8531247Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:59:24.8532449Z TIMING: _recursive_pre_grad_passes:0.01217 _recursive_joint_graph_passes:0.83459 _recursive_post_grad_passes:0.12648 async_compile.wait:0.80238 code_gen:11.22223 inductor_compile:13.69536 backend_compile:20.07298 gc:0.002 entire_frame_compile:25.90774 total_wall_time:25.90774 2025-10-10T00:59:24.8533532Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:16329 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:4866 2025-10-10T00:59:24.8534118Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-10-10T00:59:28.1195608Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T00:59:28.1196489Z import pynvml # type: ignore[import] 2025-10-10T00:59:31.6760814Z 2025-10-10T00:59:32.2267643Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:59:32.2268124Z loading model: 0it [00:00, ?it/s] 2025-10-10T00:59:32.2343648Z cpu eval MobileBertForMaskedLM 2025-10-10T00:59:32.5087430Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:59:32.6812814Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T00:59:32.8495067Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:00.3239139Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3245370Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3248505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3249031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3254014Z res = mod(**inputs) 2025-10-10T01:00:00.3256628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3257268Z outputs = self.mobilebert( 2025-10-10T01:00:00.3262033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-10-10T01:00:00.3262595Z embedding_output = self.embeddings( 2025-10-10T01:00:00.3263448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-10-10T01:00:00.3263930Z inputs_embeds = torch.cat( 2025-10-10T01:00:00.3264078Z 2025-10-10T01:00:00.3264212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3264632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3265005Z res = mod(**inputs) 2025-10-10T01:00:00.3265556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-10-10T01:00:00.3266059Z prediction_scores = self.cls(sequence_output) 2025-10-10T01:00:00.3266893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-10-10T01:00:00.3267418Z prediction_scores = self.predictions(sequence_output) 2025-10-10T01:00:00.3268048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-10-10T01:00:00.3268607Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-10-10T01:00:00.3268883Z 2025-10-10T01:00:00.3269004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3269422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3269780Z res = mod(**inputs) 2025-10-10T01:00:00.3270218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3270669Z outputs = self.mobilebert( 2025-10-10T01:00:00.3271112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-10-10T01:00:00.3271576Z embedding_output = self.embeddings( 2025-10-10T01:00:00.3272036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-10-10T01:00:00.3272541Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-10-10T01:00:00.3272734Z 2025-10-10T01:00:00.3272853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3273258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3273633Z res = mod(**inputs) 2025-10-10T01:00:00.3274059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3274517Z outputs = self.mobilebert( 2025-10-10T01:00:00.3274944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-10-10T01:00:00.3275407Z embedding_output = self.embeddings( 2025-10-10T01:00:00.3275856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-10-10T01:00:00.3276307Z embeddings = self.LayerNorm(embeddings) 2025-10-10T01:00:00.3276757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3277231Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3277405Z 2025-10-10T01:00:00.3277524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3277915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3278271Z res = mod(**inputs) 2025-10-10T01:00:00.3278685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3279138Z outputs = self.mobilebert( 2025-10-10T01:00:00.3279655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3280137Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3280606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3281067Z layer_outputs = layer_module( 2025-10-10T01:00:00.3281520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3282274Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3282818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3283303Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3283824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3284285Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3284446Z 2025-10-10T01:00:00.3284558Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3284948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3285297Z res = mod(**inputs) 2025-10-10T01:00:00.3285712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3286156Z outputs = self.mobilebert( 2025-10-10T01:00:00.3286595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3287044Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3287486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3287932Z layer_outputs = layer_module( 2025-10-10T01:00:00.3288376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3288839Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3289301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3289747Z self_outputs = self.self( 2025-10-10T01:00:00.3290199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3290659Z self.value(value_tensor) 2025-10-10T01:00:00.3290793Z 2025-10-10T01:00:00.3290921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3291325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3291677Z res = mod(**inputs) 2025-10-10T01:00:00.3292117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3292566Z outputs = self.mobilebert( 2025-10-10T01:00:00.3293011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3293470Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3293922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3294386Z layer_outputs = layer_module( 2025-10-10T01:00:00.3294854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3295430Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3295999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3296502Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3297008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3297465Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3297624Z 2025-10-10T01:00:00.3297757Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3298157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3298514Z res = mod(**inputs) 2025-10-10T01:00:00.3298945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3300165Z outputs = self.mobilebert( 2025-10-10T01:00:00.3300636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3301098Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3301553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3302001Z layer_outputs = layer_module( 2025-10-10T01:00:00.3302480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3303031Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3303587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3304089Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3304575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3305049Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3305519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3306001Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3306165Z 2025-10-10T01:00:00.3306297Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3306814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3307181Z res = mod(**inputs) 2025-10-10T01:00:00.3307612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3308074Z outputs = self.mobilebert( 2025-10-10T01:00:00.3308521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3308972Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3309427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3309885Z layer_outputs = layer_module( 2025-10-10T01:00:00.3310340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3310812Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3311271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3311724Z self_outputs = self.self( 2025-10-10T01:00:00.3312153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3312617Z self.query(query_tensor) 2025-10-10T01:00:00.3312743Z 2025-10-10T01:00:00.3312855Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3313248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3313607Z res = mod(**inputs) 2025-10-10T01:00:00.3314036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3314496Z outputs = self.mobilebert( 2025-10-10T01:00:00.3314925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3315375Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3315834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3316298Z layer_outputs = layer_module( 2025-10-10T01:00:00.3316755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3317233Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3317705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3318163Z self_outputs = self.self( 2025-10-10T01:00:00.3318615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3319049Z self.key(key_tensor) 2025-10-10T01:00:00.3319175Z 2025-10-10T01:00:00.3319269Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3319510Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3319771Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3320167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3320512Z res = mod(**inputs) 2025-10-10T01:00:00.3320939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3321389Z outputs = self.mobilebert( 2025-10-10T01:00:00.3321833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3322289Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3322757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3323220Z layer_outputs = layer_module( 2025-10-10T01:00:00.3323689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3324154Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3324617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3325129Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3325639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3326132Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3326291Z 2025-10-10T01:00:00.3326422Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3326831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3327192Z res = mod(**inputs) 2025-10-10T01:00:00.3327637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3328125Z outputs = self.mobilebert( 2025-10-10T01:00:00.3328579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3329034Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3329494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3329948Z layer_outputs = layer_module( 2025-10-10T01:00:00.3330413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3330879Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3331347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3332193Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3332752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3333319Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3333843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3334348Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3334527Z 2025-10-10T01:00:00.3334653Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3335065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3335435Z res = mod(**inputs) 2025-10-10T01:00:00.3335947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3336418Z outputs = self.mobilebert( 2025-10-10T01:00:00.3336868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3337334Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3337795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3338255Z layer_outputs = layer_module( 2025-10-10T01:00:00.3338713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3339209Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3339700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3340211Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3340718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3341199Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3341369Z 2025-10-10T01:00:00.3341491Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3341899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3342271Z res = mod(**inputs) 2025-10-10T01:00:00.3342709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3343181Z outputs = self.mobilebert( 2025-10-10T01:00:00.3343633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3344096Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3344582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3345042Z layer_outputs = layer_module( 2025-10-10T01:00:00.3345495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3345978Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3346529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3347068Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3347599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3348112Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3348312Z 2025-10-10T01:00:00.3348459Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3348860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3349200Z res = mod(**inputs) 2025-10-10T01:00:00.3349618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3350058Z outputs = self.mobilebert( 2025-10-10T01:00:00.3350490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3350934Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3351363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3351800Z layer_outputs = layer_module( 2025-10-10T01:00:00.3352233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3352702Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3353156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3353662Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3354163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3354619Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3354775Z 2025-10-10T01:00:00.3354896Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3355278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3355627Z res = mod(**inputs) 2025-10-10T01:00:00.3356050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3356519Z outputs = self.mobilebert( 2025-10-10T01:00:00.3356943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3357379Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3357812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3358261Z layer_outputs = layer_module( 2025-10-10T01:00:00.3358690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3359156Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3359618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3360160Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3360655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3361149Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3361652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3362123Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3362286Z 2025-10-10T01:00:00.3362419Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3362806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3363155Z res = mod(**inputs) 2025-10-10T01:00:00.3363567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3364046Z outputs = self.mobilebert( 2025-10-10T01:00:00.3364471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3364914Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3365358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3365814Z layer_outputs = layer_module( 2025-10-10T01:00:00.3366288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3366759Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3367246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3367762Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3368268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3368741Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3368895Z 2025-10-10T01:00:00.3369013Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3369413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3369772Z res = mod(**inputs) 2025-10-10T01:00:00.3370199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3370648Z outputs = self.mobilebert( 2025-10-10T01:00:00.3371098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3371558Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3372016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3372476Z layer_outputs = layer_module( 2025-10-10T01:00:00.3372924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3373412Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3373901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3374407Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3374903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3375404Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3375618Z 2025-10-10T01:00:00.3375741Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3376138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3376496Z res = mod(**inputs) 2025-10-10T01:00:00.3376926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3377373Z outputs = self.mobilebert( 2025-10-10T01:00:00.3377841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3378299Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3378751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3379198Z layer_outputs = layer_module( 2025-10-10T01:00:00.3379671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3380169Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3380645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3381159Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3381674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3382146Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3382308Z 2025-10-10T01:00:00.3382425Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3382824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3383205Z res = mod(**inputs) 2025-10-10T01:00:00.3383627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3384086Z outputs = self.mobilebert( 2025-10-10T01:00:00.3384558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3385018Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3385475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3385924Z layer_outputs = layer_module( 2025-10-10T01:00:00.3386445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3386948Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3387429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3388045Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3388555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3389075Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3389637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3390106Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3390269Z 2025-10-10T01:00:00.3390391Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3390783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3391130Z res = mod(**inputs) 2025-10-10T01:00:00.3391552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3392050Z outputs = self.mobilebert( 2025-10-10T01:00:00.3392488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3392944Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3393407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3393863Z layer_outputs = layer_module( 2025-10-10T01:00:00.3394339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3394820Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3395321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3395876Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3396355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3396823Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3396980Z 2025-10-10T01:00:00.3397097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3397497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3397857Z res = mod(**inputs) 2025-10-10T01:00:00.3398283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3398749Z outputs = self.mobilebert( 2025-10-10T01:00:00.3399191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3399674Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3400113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3400554Z layer_outputs = layer_module( 2025-10-10T01:00:00.3400991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3401461Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3401945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3402453Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3402943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3403439Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3403627Z 2025-10-10T01:00:00.3403741Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3404134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3404479Z res = mod(**inputs) 2025-10-10T01:00:00.3404893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3405333Z outputs = self.mobilebert( 2025-10-10T01:00:00.3405777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3406231Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3406679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3407180Z layer_outputs = layer_module( 2025-10-10T01:00:00.3407649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3408138Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3408616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3409126Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3409650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3410136Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3410302Z 2025-10-10T01:00:00.3410421Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3410828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3411195Z res = mod(**inputs) 2025-10-10T01:00:00.3411657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3412111Z outputs = self.mobilebert( 2025-10-10T01:00:00.3412559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3413021Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3413475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3413926Z layer_outputs = layer_module( 2025-10-10T01:00:00.3414373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3414852Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3415341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3415857Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3416361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3416875Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3417389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3417874Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3418041Z 2025-10-10T01:00:00.3418166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3418558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3418918Z res = mod(**inputs) 2025-10-10T01:00:00.3419363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3419823Z outputs = self.mobilebert( 2025-10-10T01:00:00.3420266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3420718Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3421171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3421626Z layer_outputs = layer_module( 2025-10-10T01:00:00.3422078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3422588Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3423097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3423595Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3423755Z 2025-10-10T01:00:00.3423884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3424282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3424635Z res = mod(**inputs) 2025-10-10T01:00:00.3425066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3425527Z outputs = self.mobilebert( 2025-10-10T01:00:00.3425997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3426536Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3426992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3427509Z layer_outputs = layer_module( 2025-10-10T01:00:00.3427964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3428476Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3428990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3429488Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3429681Z 2025-10-10T01:00:00.3429799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3430199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3430564Z res = mod(**inputs) 2025-10-10T01:00:00.3430985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3431654Z outputs = self.mobilebert( 2025-10-10T01:00:00.3432107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3432560Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3432999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3433441Z layer_outputs = layer_module( 2025-10-10T01:00:00.3433885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3434434Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3434972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3435444Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3435622Z 2025-10-10T01:00:00.3435738Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3436122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3436471Z res = mod(**inputs) 2025-10-10T01:00:00.3436888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3437330Z outputs = self.mobilebert( 2025-10-10T01:00:00.3437751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3438191Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3438627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3439064Z layer_outputs = layer_module( 2025-10-10T01:00:00.3439556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3440096Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3440634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3441134Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3441650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3442111Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3442279Z 2025-10-10T01:00:00.3442394Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3442781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3443182Z res = mod(**inputs) 2025-10-10T01:00:00.3443611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3444062Z outputs = self.mobilebert( 2025-10-10T01:00:00.3444506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3444959Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3445409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3445860Z layer_outputs = layer_module( 2025-10-10T01:00:00.3446293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3446835Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3447382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3447888Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3448393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3448856Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3449020Z 2025-10-10T01:00:00.3449138Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3449534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3449891Z res = mod(**inputs) 2025-10-10T01:00:00.3450303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3450755Z outputs = self.mobilebert( 2025-10-10T01:00:00.3451195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3451644Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3452092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3452537Z layer_outputs = layer_module( 2025-10-10T01:00:00.3452991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3453531Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3454069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3454569Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3455091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3455594Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3456095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3456581Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3456747Z 2025-10-10T01:00:00.3456879Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3457289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3457641Z res = mod(**inputs) 2025-10-10T01:00:00.3458060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3458506Z outputs = self.mobilebert( 2025-10-10T01:00:00.3458972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3459407Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3459856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3460315Z layer_outputs = layer_module( 2025-10-10T01:00:00.3460755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3461309Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3461849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3462331Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3462829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3463282Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3463432Z 2025-10-10T01:00:00.3463556Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3463950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3464309Z res = mod(**inputs) 2025-10-10T01:00:00.3464751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3465219Z outputs = self.mobilebert( 2025-10-10T01:00:00.3465664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3466132Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3466666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3467130Z layer_outputs = layer_module( 2025-10-10T01:00:00.3467584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3468042Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3468507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3468953Z self_outputs = self.self( 2025-10-10T01:00:00.3469389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3469819Z self.value(value_tensor) 2025-10-10T01:00:00.3469947Z 2025-10-10T01:00:00.3470060Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3470456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3470831Z res = mod(**inputs) 2025-10-10T01:00:00.3471242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3471677Z outputs = self.mobilebert( 2025-10-10T01:00:00.3472108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3472544Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3472996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3473439Z layer_outputs = layer_module( 2025-10-10T01:00:00.3473868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3474421Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3474979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3475473Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3475957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3476409Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3476564Z 2025-10-10T01:00:00.3476677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3477063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3477414Z res = mod(**inputs) 2025-10-10T01:00:00.3477841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3478281Z outputs = self.mobilebert( 2025-10-10T01:00:00.3478718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3479185Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3479638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3480099Z layer_outputs = layer_module( 2025-10-10T01:00:00.3480565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3481124Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3481670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3482163Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3482645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3483125Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3483600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3484091Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3484251Z 2025-10-10T01:00:00.3484372Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3484754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3485105Z res = mod(**inputs) 2025-10-10T01:00:00.3485524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3485989Z outputs = self.mobilebert( 2025-10-10T01:00:00.3486426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3486861Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3487322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3487780Z layer_outputs = layer_module( 2025-10-10T01:00:00.3488256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3488732Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3489199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3489655Z self_outputs = self.self( 2025-10-10T01:00:00.3490119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3490598Z self.query(query_tensor) 2025-10-10T01:00:00.3490731Z 2025-10-10T01:00:00.3490849Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3491251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3491607Z res = mod(**inputs) 2025-10-10T01:00:00.3492040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3492502Z outputs = self.mobilebert( 2025-10-10T01:00:00.3492945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3493405Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3493861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3494319Z layer_outputs = layer_module( 2025-10-10T01:00:00.3494772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3495240Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3495712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3496167Z self_outputs = self.self( 2025-10-10T01:00:00.3496614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3497064Z self.key(key_tensor) 2025-10-10T01:00:00.3497201Z 2025-10-10T01:00:00.3497299Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3497554Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3497824Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3498233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3498591Z res = mod(**inputs) 2025-10-10T01:00:00.3499031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3499501Z outputs = self.mobilebert( 2025-10-10T01:00:00.3499957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3500428Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3500882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3501353Z layer_outputs = layer_module( 2025-10-10T01:00:00.3501815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3502315Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3502779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3503300Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3503818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3504298Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3504476Z 2025-10-10T01:00:00.3504604Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3504991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3505348Z res = mod(**inputs) 2025-10-10T01:00:00.3505804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3506284Z outputs = self.mobilebert( 2025-10-10T01:00:00.3506812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3507292Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3507760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3508220Z layer_outputs = layer_module( 2025-10-10T01:00:00.3508681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3509156Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3509620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3510139Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3510654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3511164Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3511682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3512158Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3512340Z 2025-10-10T01:00:00.3512462Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3512877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3513250Z res = mod(**inputs) 2025-10-10T01:00:00.3513660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3514112Z outputs = self.mobilebert( 2025-10-10T01:00:00.3514553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3515002Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3515458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3515907Z layer_outputs = layer_module( 2025-10-10T01:00:00.3516359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3516859Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3517332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3517823Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3518349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3518816Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3518979Z 2025-10-10T01:00:00.3519095Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3519488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3519855Z res = mod(**inputs) 2025-10-10T01:00:00.3520301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3520774Z outputs = self.mobilebert( 2025-10-10T01:00:00.3521224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3521693Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3522153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3522635Z layer_outputs = layer_module( 2025-10-10T01:00:00.3523095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3523593Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3524079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3524597Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3525088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3525595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3525782Z 2025-10-10T01:00:00.3525911Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3526313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3526671Z res = mod(**inputs) 2025-10-10T01:00:00.3527112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3527582Z outputs = self.mobilebert( 2025-10-10T01:00:00.3528023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3528483Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3528932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3529409Z layer_outputs = layer_module( 2025-10-10T01:00:00.3529864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3530356Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3530847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3531359Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3532021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3532509Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3532670Z 2025-10-10T01:00:00.3532797Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3533204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3533564Z res = mod(**inputs) 2025-10-10T01:00:00.3534007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3534527Z outputs = self.mobilebert( 2025-10-10T01:00:00.3534970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3535421Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3535875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3536330Z layer_outputs = layer_module( 2025-10-10T01:00:00.3536803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3537294Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3537776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3538350Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3538866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3539384Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3539906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3540388Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3540565Z 2025-10-10T01:00:00.3540684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3541085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3541447Z res = mod(**inputs) 2025-10-10T01:00:00.3541900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3542354Z outputs = self.mobilebert( 2025-10-10T01:00:00.3542801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3543264Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3543717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3544173Z layer_outputs = layer_module( 2025-10-10T01:00:00.3544630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3545115Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3545601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3546118Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3546685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3547177Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3547341Z 2025-10-10T01:00:00.3547460Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3547863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3548225Z res = mod(**inputs) 2025-10-10T01:00:00.3548659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3549115Z outputs = self.mobilebert( 2025-10-10T01:00:00.3549558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3550017Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3550507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3550942Z layer_outputs = layer_module( 2025-10-10T01:00:00.3551380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3551853Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3552376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3552862Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3553338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3553825Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3554013Z 2025-10-10T01:00:00.3554161Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3554548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3554894Z res = mod(**inputs) 2025-10-10T01:00:00.3555319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3555763Z outputs = self.mobilebert( 2025-10-10T01:00:00.3556201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3556646Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3557079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3557526Z layer_outputs = layer_module( 2025-10-10T01:00:00.3557983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3558475Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3558943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3559443Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3559947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3560418Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3560575Z 2025-10-10T01:00:00.3560705Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3561103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3561457Z res = mod(**inputs) 2025-10-10T01:00:00.3561883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3562334Z outputs = self.mobilebert( 2025-10-10T01:00:00.3562778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3563231Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3563686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3564141Z layer_outputs = layer_module( 2025-10-10T01:00:00.3564587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3565073Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3565540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3566059Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3566553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3567048Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3567543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3568001Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3568185Z 2025-10-10T01:00:00.3568300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3568687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3569033Z res = mod(**inputs) 2025-10-10T01:00:00.3569472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3569957Z outputs = self.mobilebert( 2025-10-10T01:00:00.3570406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3570866Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3571313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3571771Z layer_outputs = layer_module( 2025-10-10T01:00:00.3572214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3572683Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3573152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3573654Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3574148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3574605Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3574764Z 2025-10-10T01:00:00.3574877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3575267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3575618Z res = mod(**inputs) 2025-10-10T01:00:00.3576032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3576480Z outputs = self.mobilebert( 2025-10-10T01:00:00.3576911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3577360Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3577804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3578247Z layer_outputs = layer_module( 2025-10-10T01:00:00.3578685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3579153Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3579624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3580116Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3580605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3581106Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3581320Z 2025-10-10T01:00:00.3581441Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3581835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3582191Z res = mod(**inputs) 2025-10-10T01:00:00.3582609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3583070Z outputs = self.mobilebert( 2025-10-10T01:00:00.3583525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3583993Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3584436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3584888Z layer_outputs = layer_module( 2025-10-10T01:00:00.3585352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3585855Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3586416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3586947Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3587474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3587956Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3588113Z 2025-10-10T01:00:00.3588238Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3588641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3588998Z res = mod(**inputs) 2025-10-10T01:00:00.3589437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3589900Z outputs = self.mobilebert( 2025-10-10T01:00:00.3590344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3590803Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3591249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3591707Z layer_outputs = layer_module( 2025-10-10T01:00:00.3592155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3592634Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3593113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3593633Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3594147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3594660Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3595172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3595645Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3595814Z 2025-10-10T01:00:00.3595932Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3596330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3596690Z res = mod(**inputs) 2025-10-10T01:00:00.3597120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3597614Z outputs = self.mobilebert( 2025-10-10T01:00:00.3598050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3598495Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3598936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3599382Z layer_outputs = layer_module( 2025-10-10T01:00:00.3599833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3600326Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3600812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3601304Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3601456Z 2025-10-10T01:00:00.3601577Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3601954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3602298Z res = mod(**inputs) 2025-10-10T01:00:00.3602705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3603148Z outputs = self.mobilebert( 2025-10-10T01:00:00.3603568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3604012Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3604451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3604895Z layer_outputs = layer_module( 2025-10-10T01:00:00.3605329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3605813Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3606305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3606792Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3606970Z 2025-10-10T01:00:00.3607095Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3607482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3607825Z res = mod(**inputs) 2025-10-10T01:00:00.3608240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3608687Z outputs = self.mobilebert( 2025-10-10T01:00:00.3609118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3609557Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3609990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3610429Z layer_outputs = layer_module( 2025-10-10T01:00:00.3610865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3611401Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3611930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3612426Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3612600Z 2025-10-10T01:00:00.3612716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3613106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3613452Z res = mod(**inputs) 2025-10-10T01:00:00.3613859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3614303Z outputs = self.mobilebert( 2025-10-10T01:00:00.3614749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3615192Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3615629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3616063Z layer_outputs = layer_module( 2025-10-10T01:00:00.3616539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3617076Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3617614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3618115Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3618606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3619079Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3619249Z 2025-10-10T01:00:00.3619363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3619754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3620110Z res = mod(**inputs) 2025-10-10T01:00:00.3620624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3621078Z outputs = self.mobilebert( 2025-10-10T01:00:00.3621510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3621983Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3622439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3622894Z layer_outputs = layer_module( 2025-10-10T01:00:00.3623341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3623890Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3624445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3624950Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3625458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3625928Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3626086Z 2025-10-10T01:00:00.3626211Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3626672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3627053Z res = mod(**inputs) 2025-10-10T01:00:00.3627486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3627948Z outputs = self.mobilebert( 2025-10-10T01:00:00.3628420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3628877Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3629322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3629778Z layer_outputs = layer_module( 2025-10-10T01:00:00.3630229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3630796Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3631352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3632048Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3632648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3633124Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3633624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3634107Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3634268Z 2025-10-10T01:00:00.3634385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3634800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3635170Z res = mod(**inputs) 2025-10-10T01:00:00.3635592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3636039Z outputs = self.mobilebert( 2025-10-10T01:00:00.3636487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3636940Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3637386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3637843Z layer_outputs = layer_module( 2025-10-10T01:00:00.3638280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3638832Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3639380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3639878Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3640375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3640895Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3641052Z 2025-10-10T01:00:00.3641166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3641570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3641919Z res = mod(**inputs) 2025-10-10T01:00:00.3642343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3642778Z outputs = self.mobilebert( 2025-10-10T01:00:00.3643210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3643661Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3644107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3644592Z layer_outputs = layer_module( 2025-10-10T01:00:00.3645029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3645490Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3645955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3646393Z self_outputs = self.self( 2025-10-10T01:00:00.3646851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3647321Z self.value(value_tensor) 2025-10-10T01:00:00.3647454Z 2025-10-10T01:00:00.3647568Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3647975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3648342Z res = mod(**inputs) 2025-10-10T01:00:00.3648763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3649222Z outputs = self.mobilebert( 2025-10-10T01:00:00.3649663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3650117Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3650553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3650999Z layer_outputs = layer_module( 2025-10-10T01:00:00.3651489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3652054Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3652610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3653104Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3653586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3654041Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3654190Z 2025-10-10T01:00:00.3654325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3654716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3655057Z res = mod(**inputs) 2025-10-10T01:00:00.3655474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3655925Z outputs = self.mobilebert( 2025-10-10T01:00:00.3656355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3656799Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3657231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3657674Z layer_outputs = layer_module( 2025-10-10T01:00:00.3658111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3658657Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3659217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3659693Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3660203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3660680Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3661152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3661639Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3661805Z 2025-10-10T01:00:00.3661941Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3662341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3662699Z res = mod(**inputs) 2025-10-10T01:00:00.3663125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3663577Z outputs = self.mobilebert( 2025-10-10T01:00:00.3664048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3664516Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3664971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3665429Z layer_outputs = layer_module( 2025-10-10T01:00:00.3665878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3666433Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3666931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3667399Z self_outputs = self.self( 2025-10-10T01:00:00.3667859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3668327Z self.query(query_tensor) 2025-10-10T01:00:00.3668469Z 2025-10-10T01:00:00.3668587Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3668996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3669425Z res = mod(**inputs) 2025-10-10T01:00:00.3669859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3670340Z outputs = self.mobilebert( 2025-10-10T01:00:00.3670810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3671282Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3671751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3672215Z layer_outputs = layer_module( 2025-10-10T01:00:00.3672684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3673169Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3673653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3674121Z self_outputs = self.self( 2025-10-10T01:00:00.3674569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3675035Z self.key(key_tensor) 2025-10-10T01:00:00.3675168Z 2025-10-10T01:00:00.3675263Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3675513Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3675776Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3676214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3676577Z res = mod(**inputs) 2025-10-10T01:00:00.3677008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3677468Z outputs = self.mobilebert( 2025-10-10T01:00:00.3677921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3678388Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3678830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3679271Z layer_outputs = layer_module( 2025-10-10T01:00:00.3679700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3680196Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3680652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3681145Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3681637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3682086Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3682245Z 2025-10-10T01:00:00.3682359Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3682744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3683089Z res = mod(**inputs) 2025-10-10T01:00:00.3683498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3683935Z outputs = self.mobilebert( 2025-10-10T01:00:00.3684375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3684823Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3685264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3685701Z layer_outputs = layer_module( 2025-10-10T01:00:00.3686130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3686589Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3687050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3687560Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3688079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3688571Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3689068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3689530Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3689689Z 2025-10-10T01:00:00.3689810Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3690184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3690528Z res = mod(**inputs) 2025-10-10T01:00:00.3690945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3691430Z outputs = self.mobilebert( 2025-10-10T01:00:00.3691874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3692320Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3692762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3693223Z layer_outputs = layer_module( 2025-10-10T01:00:00.3693709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3694178Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3694641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3695130Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3695669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3696143Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3696300Z 2025-10-10T01:00:00.3696424Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3696817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3697177Z res = mod(**inputs) 2025-10-10T01:00:00.3697607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3698066Z outputs = self.mobilebert( 2025-10-10T01:00:00.3698497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3698956Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3699415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3699874Z layer_outputs = layer_module( 2025-10-10T01:00:00.3700323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3700797Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3701282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3701801Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3702301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3702802Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3702985Z 2025-10-10T01:00:00.3703105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3703506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3703863Z res = mod(**inputs) 2025-10-10T01:00:00.3704287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3704745Z outputs = self.mobilebert( 2025-10-10T01:00:00.3705179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3705641Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3706091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3706616Z layer_outputs = layer_module( 2025-10-10T01:00:00.3707065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3707575Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3708070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3708590Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3709115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3709594Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3709786Z 2025-10-10T01:00:00.3709906Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3710307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3710659Z res = mod(**inputs) 2025-10-10T01:00:00.3711105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3711577Z outputs = self.mobilebert( 2025-10-10T01:00:00.3712031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3712498Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3712960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3713415Z layer_outputs = layer_module( 2025-10-10T01:00:00.3713853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3714320Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3714799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3716851Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3717355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3717848Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3718349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3718819Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3718982Z 2025-10-10T01:00:00.3719107Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3719324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3719404Z res = mod(**inputs) 2025-10-10T01:00:00.3719702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3719786Z outputs = self.mobilebert( 2025-10-10T01:00:00.3720096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3720177Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3720483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3720562Z layer_outputs = layer_module( 2025-10-10T01:00:00.3720871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3720986Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3721294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3721430Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3721772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3721878Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3721882Z 2025-10-10T01:00:00.3722000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3722217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3722300Z res = mod(**inputs) 2025-10-10T01:00:00.3722629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3722718Z outputs = self.mobilebert( 2025-10-10T01:00:00.3723016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3723094Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3723469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3723550Z layer_outputs = layer_module( 2025-10-10T01:00:00.3723861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3723967Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3724281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3724409Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3724726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3724857Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3724863Z 2025-10-10T01:00:00.3724979Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3725201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3725271Z res = mod(**inputs) 2025-10-10T01:00:00.3725570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3725655Z outputs = self.mobilebert( 2025-10-10T01:00:00.3725953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3726041Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3726339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3726424Z layer_outputs = layer_module( 2025-10-10T01:00:00.3726725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3726831Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3727139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3727274Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3727587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3727682Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3727686Z 2025-10-10T01:00:00.3727808Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3728027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3728100Z res = mod(**inputs) 2025-10-10T01:00:00.3728423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3728527Z outputs = self.mobilebert( 2025-10-10T01:00:00.3728845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3728927Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3729243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3729332Z layer_outputs = layer_module( 2025-10-10T01:00:00.3729674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3729788Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3730088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3730289Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3730602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3730737Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3731056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3731163Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3731169Z 2025-10-10T01:00:00.3731292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3731725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3731805Z res = mod(**inputs) 2025-10-10T01:00:00.3732140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3732224Z outputs = self.mobilebert( 2025-10-10T01:00:00.3732560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3732642Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3732978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3733058Z layer_outputs = layer_module( 2025-10-10T01:00:00.3733382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3733499Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3733822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3733961Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3734276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3734370Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3734374Z 2025-10-10T01:00:00.3734502Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3734720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3734805Z res = mod(**inputs) 2025-10-10T01:00:00.3735130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3735217Z outputs = self.mobilebert( 2025-10-10T01:00:00.3735541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3735697Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3736019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3736100Z layer_outputs = layer_module( 2025-10-10T01:00:00.3736414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3736518Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3736858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3736995Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3737306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3737440Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3737474Z 2025-10-10T01:00:00.3737630Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3737856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3737928Z res = mod(**inputs) 2025-10-10T01:00:00.3738235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3738324Z outputs = self.mobilebert( 2025-10-10T01:00:00.3738631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3738720Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3739028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3739107Z layer_outputs = layer_module( 2025-10-10T01:00:00.3739425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3739530Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3739841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3739980Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3740298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3740395Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3740399Z 2025-10-10T01:00:00.3740514Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3740741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3740812Z res = mod(**inputs) 2025-10-10T01:00:00.3741134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3741215Z outputs = self.mobilebert( 2025-10-10T01:00:00.3741524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3741613Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3741922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3742010Z layer_outputs = layer_module( 2025-10-10T01:00:00.3742317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3742430Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3742741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3742914Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3743230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3743366Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3743677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3743800Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3743804Z 2025-10-10T01:00:00.3743928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3744144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3744217Z res = mod(**inputs) 2025-10-10T01:00:00.3744549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3744650Z outputs = self.mobilebert( 2025-10-10T01:00:00.3744971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3745052Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3745360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3745451Z layer_outputs = layer_module( 2025-10-10T01:00:00.3745763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3745906Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3746219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3746413Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3746446Z 2025-10-10T01:00:00.3746583Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3746801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3746886Z res = mod(**inputs) 2025-10-10T01:00:00.3747207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3747296Z outputs = self.mobilebert( 2025-10-10T01:00:00.3747625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3747705Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3748015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3748098Z layer_outputs = layer_module( 2025-10-10T01:00:00.3748407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3748538Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3748849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3748979Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3748983Z 2025-10-10T01:00:00.3749096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3749315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3749385Z res = mod(**inputs) 2025-10-10T01:00:00.3749706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3749816Z outputs = self.mobilebert( 2025-10-10T01:00:00.3750116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3750202Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3750503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3750588Z layer_outputs = layer_module( 2025-10-10T01:00:00.3750909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3751080Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3751391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3751493Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3751531Z 2025-10-10T01:00:00.3751650Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3751862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3751938Z res = mod(**inputs) 2025-10-10T01:00:00.3752247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3752323Z outputs = self.mobilebert( 2025-10-10T01:00:00.3752644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3752722Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3753026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3753103Z layer_outputs = layer_module( 2025-10-10T01:00:00.3753409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3753586Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3753885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3754030Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3754342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3754455Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3754459Z 2025-10-10T01:00:00.3754581Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3754791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3754873Z res = mod(**inputs) 2025-10-10T01:00:00.3755170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3755255Z outputs = self.mobilebert( 2025-10-10T01:00:00.3755564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3755649Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3755960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3756036Z layer_outputs = layer_module( 2025-10-10T01:00:00.3756340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3756506Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3756859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3756994Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3757300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3757403Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3757407Z 2025-10-10T01:00:00.3757519Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3757770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3757845Z res = mod(**inputs) 2025-10-10T01:00:00.3758157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3758235Z outputs = self.mobilebert( 2025-10-10T01:00:00.3758590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3758677Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3758977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3759062Z layer_outputs = layer_module( 2025-10-10T01:00:00.3759362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3759532Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3759845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3759983Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3760304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3760440Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3760752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3760857Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3760861Z 2025-10-10T01:00:00.3760977Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3761207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3761282Z res = mod(**inputs) 2025-10-10T01:00:00.3761599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3761678Z outputs = self.mobilebert( 2025-10-10T01:00:00.3762004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3762086Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3762400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3762485Z layer_outputs = layer_module( 2025-10-10T01:00:00.3762793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3762986Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3763295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3763423Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3763759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3763856Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3763860Z 2025-10-10T01:00:00.3763984Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3764201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3764282Z res = mod(**inputs) 2025-10-10T01:00:00.3764603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3764685Z outputs = self.mobilebert( 2025-10-10T01:00:00.3764997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3765078Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3765412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3765509Z layer_outputs = layer_module( 2025-10-10T01:00:00.3765818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3765926Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3766232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3766321Z self_outputs = self.self( 2025-10-10T01:00:00.3766631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3766720Z self.value(value_tensor) 2025-10-10T01:00:00.3766724Z 2025-10-10T01:00:00.3766840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3767060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3767144Z res = mod(**inputs) 2025-10-10T01:00:00.3767452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3767535Z outputs = self.mobilebert( 2025-10-10T01:00:00.3767842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3767922Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3768240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3768321Z layer_outputs = layer_module( 2025-10-10T01:00:00.3768637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3768815Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3769131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3769254Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3769572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3769671Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3769675Z 2025-10-10T01:00:00.3769793Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3770015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3770088Z res = mod(**inputs) 2025-10-10T01:00:00.3770392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3770504Z outputs = self.mobilebert( 2025-10-10T01:00:00.3770809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3770898Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3771223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3771310Z layer_outputs = layer_module( 2025-10-10T01:00:00.3771647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3771819Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3772123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3772242Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3772587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3772685Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3772983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3773091Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3773095Z 2025-10-10T01:00:00.3773207Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3773429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3773500Z res = mod(**inputs) 2025-10-10T01:00:00.3773815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3773894Z outputs = self.mobilebert( 2025-10-10T01:00:00.3774194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3774282Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3774580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3774664Z layer_outputs = layer_module( 2025-10-10T01:00:00.3774974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3775068Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3775383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3775460Z self_outputs = self.self( 2025-10-10T01:00:00.3775770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3775851Z self.query(query_tensor) 2025-10-10T01:00:00.3775855Z 2025-10-10T01:00:00.3775975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3776187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3776258Z res = mod(**inputs) 2025-10-10T01:00:00.3776563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3776642Z outputs = self.mobilebert( 2025-10-10T01:00:00.3776959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3777037Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3777346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3777461Z layer_outputs = layer_module( 2025-10-10T01:00:00.3777762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3777858Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3778168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3778251Z self_outputs = self.self( 2025-10-10T01:00:00.3778572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3778648Z self.key(key_tensor) 2025-10-10T01:00:00.3778652Z 2025-10-10T01:00:00.3778752Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3778840Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3778960Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3779206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3779277Z res = mod(**inputs) 2025-10-10T01:00:00.3779579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3779654Z outputs = self.mobilebert( 2025-10-10T01:00:00.3779958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3780039Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3780337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3780421Z layer_outputs = layer_module( 2025-10-10T01:00:00.3780721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3780828Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3781128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3781270Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3781567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3781659Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3781663Z 2025-10-10T01:00:00.3781784Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3781994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3782072Z res = mod(**inputs) 2025-10-10T01:00:00.3782369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3782450Z outputs = self.mobilebert( 2025-10-10T01:00:00.3782755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3782835Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3783142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3783221Z layer_outputs = layer_module( 2025-10-10T01:00:00.3783535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3783629Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3783935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3784078Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3784411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3784560Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3784868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3784979Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3784983Z 2025-10-10T01:00:00.3785115Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3785331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3785412Z res = mod(**inputs) 2025-10-10T01:00:00.3785719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3785835Z outputs = self.mobilebert( 2025-10-10T01:00:00.3786160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3786242Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3786637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3786721Z layer_outputs = layer_module( 2025-10-10T01:00:00.3787039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3787145Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3787458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3787584Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3787896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3787998Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3788002Z 2025-10-10T01:00:00.3788116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3788339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3788412Z res = mod(**inputs) 2025-10-10T01:00:00.3788718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3788806Z outputs = self.mobilebert( 2025-10-10T01:00:00.3789113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3789203Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3789510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3789601Z layer_outputs = layer_module( 2025-10-10T01:00:00.3789909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3790013Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3790332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3790457Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3790777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3790901Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3790905Z 2025-10-10T01:00:00.3791019Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3791269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3791342Z res = mod(**inputs) 2025-10-10T01:00:00.3791662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3791742Z outputs = self.mobilebert( 2025-10-10T01:00:00.3792061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3792163Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3792470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3792556Z layer_outputs = layer_module( 2025-10-10T01:00:00.3792875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3793023Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3793334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3793473Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3793798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3793892Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3793897Z 2025-10-10T01:00:00.3794018Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3794236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3794315Z res = mod(**inputs) 2025-10-10T01:00:00.3794639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3794717Z outputs = self.mobilebert( 2025-10-10T01:00:00.3795025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3795102Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3795415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3795492Z layer_outputs = layer_module( 2025-10-10T01:00:00.3795792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3795899Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3796210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3796357Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3796659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3796797Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3797103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3797204Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3797207Z 2025-10-10T01:00:00.3797328Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3797540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3797615Z res = mod(**inputs) 2025-10-10T01:00:00.3797926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3798054Z outputs = self.mobilebert( 2025-10-10T01:00:00.3798360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3798437Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3798747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3798823Z layer_outputs = layer_module( 2025-10-10T01:00:00.3799147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3799245Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3799522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3799645Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3799960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3800058Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3800061Z 2025-10-10T01:00:00.3800166Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3800372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3800439Z res = mod(**inputs) 2025-10-10T01:00:00.3800723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3800806Z outputs = self.mobilebert( 2025-10-10T01:00:00.3801084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3801167Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3801450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3801524Z layer_outputs = layer_module( 2025-10-10T01:00:00.3801812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3801912Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3802216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3802339Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3802636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3802781Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3802785Z 2025-10-10T01:00:00.3802901Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3803118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3803188Z res = mod(**inputs) 2025-10-10T01:00:00.3803491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3803568Z outputs = self.mobilebert( 2025-10-10T01:00:00.3803870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3803953Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3804230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3804312Z layer_outputs = layer_module( 2025-10-10T01:00:00.3804594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3804713Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3804999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3805133Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3805436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3806146Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3806151Z 2025-10-10T01:00:00.3806301Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3806516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3806587Z res = mod(**inputs) 2025-10-10T01:00:00.3806919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3807015Z outputs = self.mobilebert( 2025-10-10T01:00:00.3807331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3807408Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3807719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3807797Z layer_outputs = layer_module( 2025-10-10T01:00:00.3808105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3808215Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3808520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3808665Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3808969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3809100Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3809413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3809513Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3809519Z 2025-10-10T01:00:00.3809635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3809849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3809927Z res = mod(**inputs) 2025-10-10T01:00:00.3810231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3810310Z outputs = self.mobilebert( 2025-10-10T01:00:00.3810620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3810696Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3811007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3811084Z layer_outputs = layer_module( 2025-10-10T01:00:00.3811389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3811494Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3811797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3811948Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3812249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3812347Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3812351Z 2025-10-10T01:00:00.3812463Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3812674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3812751Z res = mod(**inputs) 2025-10-10T01:00:00.3813075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3813163Z outputs = self.mobilebert( 2025-10-10T01:00:00.3813462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3813544Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3813878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3813958Z layer_outputs = layer_module( 2025-10-10T01:00:00.3814258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3814358Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3814661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3814781Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3815075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3815201Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3815207Z 2025-10-10T01:00:00.3815320Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3815536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3815606Z res = mod(**inputs) 2025-10-10T01:00:00.3815903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3815987Z outputs = self.mobilebert( 2025-10-10T01:00:00.3816293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3816382Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3816685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3816772Z layer_outputs = layer_module( 2025-10-10T01:00:00.3817076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3817184Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3817498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3817634Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3817948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3818054Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3818057Z 2025-10-10T01:00:00.3818176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3818385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3818456Z res = mod(**inputs) 2025-10-10T01:00:00.3818781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3818859Z outputs = self.mobilebert( 2025-10-10T01:00:00.3819163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3819239Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3819539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3819641Z layer_outputs = layer_module( 2025-10-10T01:00:00.3819943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3820052Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3820371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3820520Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3820824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3820955Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3821261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3821363Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3821367Z 2025-10-10T01:00:00.3821484Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3821696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3821766Z res = mod(**inputs) 2025-10-10T01:00:00.3822073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3822152Z outputs = self.mobilebert( 2025-10-10T01:00:00.3822456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3822535Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3822839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3822916Z layer_outputs = layer_module( 2025-10-10T01:00:00.3823216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3823355Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3823651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3823754Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3823758Z 2025-10-10T01:00:00.3823869Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3824083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3824161Z res = mod(**inputs) 2025-10-10T01:00:00.3824460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3824544Z outputs = self.mobilebert( 2025-10-10T01:00:00.3824842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3824928Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3825225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3825329Z layer_outputs = layer_module( 2025-10-10T01:00:00.3825645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3825778Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3826092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3826213Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3826217Z 2025-10-10T01:00:00.3826597Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3826833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3826907Z res = mod(**inputs) 2025-10-10T01:00:00.3827223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3827348Z outputs = self.mobilebert( 2025-10-10T01:00:00.3827666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3827748Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3828059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3828144Z layer_outputs = layer_module( 2025-10-10T01:00:00.3828449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3828635Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3828940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3829045Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3829064Z 2025-10-10T01:00:00.3829179Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3829396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3829476Z res = mod(**inputs) 2025-10-10T01:00:00.3829780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3829864Z outputs = self.mobilebert( 2025-10-10T01:00:00.3830173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3830255Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3830568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3830648Z layer_outputs = layer_module( 2025-10-10T01:00:00.3830964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3831137Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3831703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3831860Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3832177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3832291Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3832299Z 2025-10-10T01:00:00.3832415Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3832642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3832771Z res = mod(**inputs) 2025-10-10T01:00:00.3833078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3833167Z outputs = self.mobilebert( 2025-10-10T01:00:00.3833469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3833559Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3833896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3833979Z layer_outputs = layer_module( 2025-10-10T01:00:00.3834305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3834476Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3834843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3834983Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3835313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3835406Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3835410Z 2025-10-10T01:00:00.3835524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3835753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3835826Z res = mod(**inputs) 2025-10-10T01:00:00.3836152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3836230Z outputs = self.mobilebert( 2025-10-10T01:00:00.3836551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3836632Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3836944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3837031Z layer_outputs = layer_module( 2025-10-10T01:00:00.3837339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3837516Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3837829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3837953Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3838249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3838375Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3838667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3838761Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3838765Z 2025-10-10T01:00:00.3838876Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3839076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3839143Z res = mod(**inputs) 2025-10-10T01:00:00.3839432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3839504Z outputs = self.mobilebert( 2025-10-10T01:00:00.3839820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3839897Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3840176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3840258Z layer_outputs = layer_module( 2025-10-10T01:00:00.3840554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3840749Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3841049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3841177Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3841489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3841596Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3841600Z 2025-10-10T01:00:00.3841716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3841928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3842004Z res = mod(**inputs) 2025-10-10T01:00:00.3842303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3842380Z outputs = self.mobilebert( 2025-10-10T01:00:00.3842683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3842762Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3843067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3843147Z layer_outputs = layer_module( 2025-10-10T01:00:00.3843459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3843547Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3843829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3843911Z self_outputs = self.self( 2025-10-10T01:00:00.3844191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3844271Z self.value(value_tensor) 2025-10-10T01:00:00.3844274Z 2025-10-10T01:00:00.3844378Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3844577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3844656Z res = mod(**inputs) 2025-10-10T01:00:00.3844936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3845013Z outputs = self.mobilebert( 2025-10-10T01:00:00.3845301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3845387Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3845685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3845762Z layer_outputs = layer_module( 2025-10-10T01:00:00.3846063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3846233Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3846567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3846686Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3846982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3847076Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3847081Z 2025-10-10T01:00:00.3847207Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3847428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3847499Z res = mod(**inputs) 2025-10-10T01:00:00.3847808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3847922Z outputs = self.mobilebert( 2025-10-10T01:00:00.3848218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3848303Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3848598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3848683Z layer_outputs = layer_module( 2025-10-10T01:00:00.3848979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3849147Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3849454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3849572Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3849879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3849974Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3850278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3850379Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3850383Z 2025-10-10T01:00:00.3850494Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3850713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3850783Z res = mod(**inputs) 2025-10-10T01:00:00.3851089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3851165Z outputs = self.mobilebert( 2025-10-10T01:00:00.3851476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3851555Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3851851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3851937Z layer_outputs = layer_module( 2025-10-10T01:00:00.3852246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3852348Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3852655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3852733Z self_outputs = self.self( 2025-10-10T01:00:00.3853048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3853149Z self.query(query_tensor) 2025-10-10T01:00:00.3853153Z 2025-10-10T01:00:00.3853272Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3853485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3853563Z res = mod(**inputs) 2025-10-10T01:00:00.3853879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3853972Z outputs = self.mobilebert( 2025-10-10T01:00:00.3854290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3854369Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3854690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3854798Z layer_outputs = layer_module( 2025-10-10T01:00:00.3855080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3855173Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3855480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3855564Z self_outputs = self.self( 2025-10-10T01:00:00.3855864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3855936Z self.key(key_tensor) 2025-10-10T01:00:00.3855948Z 2025-10-10T01:00:00.3856039Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3856127Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3856243Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3856457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3856534Z res = mod(**inputs) 2025-10-10T01:00:00.3856836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3856913Z outputs = self.mobilebert( 2025-10-10T01:00:00.3857220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3857299Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3857608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3857684Z layer_outputs = layer_module( 2025-10-10T01:00:00.3857991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3858096Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3858395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3858533Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3858850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3858951Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3858955Z 2025-10-10T01:00:00.3859066Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3859278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3859356Z res = mod(**inputs) 2025-10-10T01:00:00.3859665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3859772Z outputs = self.mobilebert( 2025-10-10T01:00:00.3860073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3860153Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3860461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3860539Z layer_outputs = layer_module( 2025-10-10T01:00:00.3860861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3860953Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3861265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3861403Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3861783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3861934Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3862244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3862351Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3862355Z 2025-10-10T01:00:00.3862465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3862676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3862751Z res = mod(**inputs) 2025-10-10T01:00:00.3863049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3863134Z outputs = self.mobilebert( 2025-10-10T01:00:00.3863434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3863520Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3863818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3863896Z layer_outputs = layer_module( 2025-10-10T01:00:00.3864204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3864306Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3864610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3864732Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3865042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3865143Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3865147Z 2025-10-10T01:00:00.3865259Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3865484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3865557Z res = mod(**inputs) 2025-10-10T01:00:00.3865870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3865949Z outputs = self.mobilebert( 2025-10-10T01:00:00.3866254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3866409Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3866725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3866837Z layer_outputs = layer_module( 2025-10-10T01:00:00.3867144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3867249Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3867575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3867713Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3868023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3868146Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3868151Z 2025-10-10T01:00:00.3868271Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3868516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3868588Z res = mod(**inputs) 2025-10-10T01:00:00.3868891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3868966Z outputs = self.mobilebert( 2025-10-10T01:00:00.3869257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3869332Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3869623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3869712Z layer_outputs = layer_module( 2025-10-10T01:00:00.3870008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3870122Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3870418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3870563Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3870860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3870952Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3870957Z 2025-10-10T01:00:00.3871075Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3871285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3871370Z res = mod(**inputs) 2025-10-10T01:00:00.3871652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3871726Z outputs = self.mobilebert( 2025-10-10T01:00:00.3872014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3872086Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3872371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3872452Z layer_outputs = layer_module( 2025-10-10T01:00:00.3872740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3872835Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3873120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3873280Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3873579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3873716Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3874018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3874124Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3874128Z 2025-10-10T01:00:00.3874253Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3874466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3874545Z res = mod(**inputs) 2025-10-10T01:00:00.3874851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3874970Z outputs = self.mobilebert( 2025-10-10T01:00:00.3875265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3875341Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3875644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3875722Z layer_outputs = layer_module( 2025-10-10T01:00:00.3876033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3876134Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3876446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3876568Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3876871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3876968Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3876972Z 2025-10-10T01:00:00.3877088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3877296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3877363Z res = mod(**inputs) 2025-10-10T01:00:00.3877646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3877725Z outputs = self.mobilebert( 2025-10-10T01:00:00.3878009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3878088Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3878373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3878452Z layer_outputs = layer_module( 2025-10-10T01:00:00.3878731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3878828Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3879132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3879254Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3879554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3879675Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3879678Z 2025-10-10T01:00:00.3879808Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3880026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3880096Z res = mod(**inputs) 2025-10-10T01:00:00.3880400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3880477Z outputs = self.mobilebert( 2025-10-10T01:00:00.3880790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3880882Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3881171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3881249Z layer_outputs = layer_module( 2025-10-10T01:00:00.3881546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3881667Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3881950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3882076Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3882368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3882455Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3882459Z 2025-10-10T01:00:00.3882568Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3882771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3882848Z res = mod(**inputs) 2025-10-10T01:00:00.3883149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3883229Z outputs = self.mobilebert( 2025-10-10T01:00:00.3883539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3883614Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3883902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3883977Z layer_outputs = layer_module( 2025-10-10T01:00:00.3884257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3884359Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3884644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3884783Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3885067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3885197Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3885479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3885575Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3885579Z 2025-10-10T01:00:00.3885693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3885892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3885965Z res = mod(**inputs) 2025-10-10T01:00:00.3886248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3886344Z outputs = self.mobilebert( 2025-10-10T01:00:00.3886648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3886727Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3887028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3887105Z layer_outputs = layer_module( 2025-10-10T01:00:00.3887425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3887529Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3887828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3887983Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3888298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3888396Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3888400Z 2025-10-10T01:00:00.3888510Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3888728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3888800Z res = mod(**inputs) 2025-10-10T01:00:00.3889098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3889184Z outputs = self.mobilebert( 2025-10-10T01:00:00.3889477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3889561Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3889865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3889941Z layer_outputs = layer_module( 2025-10-10T01:00:00.3890247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3890349Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3890660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3890783Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3891088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3891220Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3891227Z 2025-10-10T01:00:00.3891344Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3891575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3891645Z res = mod(**inputs) 2025-10-10T01:00:00.3891950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3892027Z outputs = self.mobilebert( 2025-10-10T01:00:00.3892321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3892407Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3892701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3892785Z layer_outputs = layer_module( 2025-10-10T01:00:00.3893080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3893198Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3893505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3893642Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3893955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3894064Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3894068Z 2025-10-10T01:00:00.3894188Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3894399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3894469Z res = mod(**inputs) 2025-10-10T01:00:00.3894790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3894882Z outputs = self.mobilebert( 2025-10-10T01:00:00.3895191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3895269Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3895587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3895667Z layer_outputs = layer_module( 2025-10-10T01:00:00.3895982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3896092Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3896405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3896553Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3896856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3896990Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3897319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3897424Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3897427Z 2025-10-10T01:00:00.3897548Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3897766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3897845Z res = mod(**inputs) 2025-10-10T01:00:00.3898170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3898251Z outputs = self.mobilebert( 2025-10-10T01:00:00.3898567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3898647Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3898977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3899057Z layer_outputs = layer_module( 2025-10-10T01:00:00.3899382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3899526Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3899850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3899975Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3899979Z 2025-10-10T01:00:00.3900091Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3900312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3900383Z res = mod(**inputs) 2025-10-10T01:00:00.3900700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3900788Z outputs = self.mobilebert( 2025-10-10T01:00:00.3901117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3901207Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3901520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3901601Z layer_outputs = layer_module( 2025-10-10T01:00:00.3901950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3902083Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3902397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3902518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3902523Z 2025-10-10T01:00:00.3902647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3902865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3902937Z res = mod(**inputs) 2025-10-10T01:00:00.3903253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3903337Z outputs = self.mobilebert( 2025-10-10T01:00:00.3903652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3903733Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3904046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3904132Z layer_outputs = layer_module( 2025-10-10T01:00:00.3904442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3904624Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3904933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3905046Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3905054Z 2025-10-10T01:00:00.3905167Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3905387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3905467Z res = mod(**inputs) 2025-10-10T01:00:00.3905776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3905861Z outputs = self.mobilebert( 2025-10-10T01:00:00.3906170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3906251Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3906641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3906727Z layer_outputs = layer_module( 2025-10-10T01:00:00.3907068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3907246Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3907565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3907702Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3908028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3908144Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3908148Z 2025-10-10T01:00:00.3908263Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3908487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3908579Z res = mod(**inputs) 2025-10-10T01:00:00.3908912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3908992Z outputs = self.mobilebert( 2025-10-10T01:00:00.3909301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3909389Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3909699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3909787Z layer_outputs = layer_module( 2025-10-10T01:00:00.3910094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3910265Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3910582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3910719Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3911032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3911126Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3911131Z 2025-10-10T01:00:00.3911250Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3911466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3911538Z res = mod(**inputs) 2025-10-10T01:00:00.3911849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3911926Z outputs = self.mobilebert( 2025-10-10T01:00:00.3912241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3912321Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3912626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3912713Z layer_outputs = layer_module( 2025-10-10T01:00:00.3913020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3913199Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3913505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3913647Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3913974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3914110Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3914422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3914524Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3914528Z 2025-10-10T01:00:00.3914646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3914888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3914961Z res = mod(**inputs) 2025-10-10T01:00:00.3915274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3915351Z outputs = self.mobilebert( 2025-10-10T01:00:00.3915682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3915778Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3916089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3916168Z layer_outputs = layer_module( 2025-10-10T01:00:00.3916473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3916659Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3916968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3917097Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3917405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3917506Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3917510Z 2025-10-10T01:00:00.3917626Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3917841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3917922Z res = mod(**inputs) 2025-10-10T01:00:00.3918227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3918311Z outputs = self.mobilebert( 2025-10-10T01:00:00.3918618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3918697Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3919011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3919095Z layer_outputs = layer_module( 2025-10-10T01:00:00.3919409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3919505Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3919819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3919900Z self_outputs = self.self( 2025-10-10T01:00:00.3920214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3920302Z self.value(value_tensor) 2025-10-10T01:00:00.3920305Z 2025-10-10T01:00:00.3920417Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3920637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3920728Z res = mod(**inputs) 2025-10-10T01:00:00.3921027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3921112Z outputs = self.mobilebert( 2025-10-10T01:00:00.3921412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3921498Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3921813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3921893Z layer_outputs = layer_module( 2025-10-10T01:00:00.3922199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3922388Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3922720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3922843Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3923154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3923243Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3923247Z 2025-10-10T01:00:00.3923360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3923582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3923651Z res = mod(**inputs) 2025-10-10T01:00:00.3923959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3924037Z outputs = self.mobilebert( 2025-10-10T01:00:00.3924353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3924429Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3924730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3924816Z layer_outputs = layer_module( 2025-10-10T01:00:00.3925118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3925293Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3925598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3925714Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3926026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.3926119Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.3926428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3926528Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3926531Z 2025-10-10T01:00:00.3926648Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3926861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3926932Z res = mod(**inputs) 2025-10-10T01:00:00.3927254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3927330Z outputs = self.mobilebert( 2025-10-10T01:00:00.3927661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3927742Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3928048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3928135Z layer_outputs = layer_module( 2025-10-10T01:00:00.3928443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3928561Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3928860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3928944Z self_outputs = self.self( 2025-10-10T01:00:00.3929264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.3929359Z self.query(query_tensor) 2025-10-10T01:00:00.3929363Z 2025-10-10T01:00:00.3929483Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3929692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3929769Z res = mod(**inputs) 2025-10-10T01:00:00.3930065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3930142Z outputs = self.mobilebert( 2025-10-10T01:00:00.3930445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3930523Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3930828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3930908Z layer_outputs = layer_module( 2025-10-10T01:00:00.3931212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3931303Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3931807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3931899Z self_outputs = self.self( 2025-10-10T01:00:00.3932199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.3932280Z self.key(key_tensor) 2025-10-10T01:00:00.3932284Z 2025-10-10T01:00:00.3932373Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3932460Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.3932583Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3932797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3932876Z res = mod(**inputs) 2025-10-10T01:00:00.3933176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3933253Z outputs = self.mobilebert( 2025-10-10T01:00:00.3933558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3933638Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3933942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3934020Z layer_outputs = layer_module( 2025-10-10T01:00:00.3934324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3934466Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3934763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3934905Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3935205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.3935306Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3935310Z 2025-10-10T01:00:00.3935442Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3935655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3935730Z res = mod(**inputs) 2025-10-10T01:00:00.3936027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3936164Z outputs = self.mobilebert( 2025-10-10T01:00:00.3936468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3936560Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3936878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3936958Z layer_outputs = layer_module( 2025-10-10T01:00:00.3937275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3937371Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3937695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.3937828Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.3938130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.3938274Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3938568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3938679Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3938683Z 2025-10-10T01:00:00.3938796Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3939016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3939085Z res = mod(**inputs) 2025-10-10T01:00:00.3939392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3939480Z outputs = self.mobilebert( 2025-10-10T01:00:00.3939783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3939871Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3940168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3940246Z layer_outputs = layer_module( 2025-10-10T01:00:00.3940562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3940665Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3940968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3941092Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3941430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3941524Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3941528Z 2025-10-10T01:00:00.3941650Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3941868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3941937Z res = mod(**inputs) 2025-10-10T01:00:00.3942260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3942338Z outputs = self.mobilebert( 2025-10-10T01:00:00.3942642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3942729Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3943054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3943157Z layer_outputs = layer_module( 2025-10-10T01:00:00.3943468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3943578Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3943899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3944023Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3944342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3944465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3944469Z 2025-10-10T01:00:00.3944588Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3944810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3944886Z res = mod(**inputs) 2025-10-10T01:00:00.3945199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3945277Z outputs = self.mobilebert( 2025-10-10T01:00:00.3945602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3945682Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3946020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3946100Z layer_outputs = layer_module( 2025-10-10T01:00:00.3946500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3946635Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3946942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3947091Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3947398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3947499Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3947505Z 2025-10-10T01:00:00.3947621Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3947838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3947930Z res = mod(**inputs) 2025-10-10T01:00:00.3948228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3948341Z outputs = self.mobilebert( 2025-10-10T01:00:00.3948645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3948726Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3949040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3949119Z layer_outputs = layer_module( 2025-10-10T01:00:00.3949448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3949554Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3949869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3950026Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3950350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3950492Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3950793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3950901Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3950905Z 2025-10-10T01:00:00.3951019Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3951234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3951314Z res = mod(**inputs) 2025-10-10T01:00:00.3951618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3951708Z outputs = self.mobilebert( 2025-10-10T01:00:00.3952015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3952102Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3952407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3952486Z layer_outputs = layer_module( 2025-10-10T01:00:00.3952800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3952905Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3953216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3953340Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3953653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3953753Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3953757Z 2025-10-10T01:00:00.3953868Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3954093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3954165Z res = mod(**inputs) 2025-10-10T01:00:00.3954478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3954558Z outputs = self.mobilebert( 2025-10-10T01:00:00.3954863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3954951Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3955280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3955371Z layer_outputs = layer_module( 2025-10-10T01:00:00.3955683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3955786Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3956103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3956244Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3956559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3956685Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3956689Z 2025-10-10T01:00:00.3956829Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3957064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3957138Z res = mod(**inputs) 2025-10-10T01:00:00.3957453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3957542Z outputs = self.mobilebert( 2025-10-10T01:00:00.3957857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3957938Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3958246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3958327Z layer_outputs = layer_module( 2025-10-10T01:00:00.3958611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3958717Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3958998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3959133Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3959416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3959505Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3959508Z 2025-10-10T01:00:00.3959620Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3959821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3959896Z res = mod(**inputs) 2025-10-10T01:00:00.3960181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3960256Z outputs = self.mobilebert( 2025-10-10T01:00:00.3960546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3960622Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3960911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3960986Z layer_outputs = layer_module( 2025-10-10T01:00:00.3961277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3961373Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3961652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3961815Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3962096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3962228Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3962512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3962613Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3962617Z 2025-10-10T01:00:00.3962739Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3962937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3963011Z res = mod(**inputs) 2025-10-10T01:00:00.3963293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3963423Z outputs = self.mobilebert( 2025-10-10T01:00:00.3963707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3963786Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3964099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3964175Z layer_outputs = layer_module( 2025-10-10T01:00:00.3964485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3964579Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3964865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3964979Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3965263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3965355Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3965358Z 2025-10-10T01:00:00.3965463Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3965675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3965745Z res = mod(**inputs) 2025-10-10T01:00:00.3966053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3966137Z outputs = self.mobilebert( 2025-10-10T01:00:00.3966443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3966526Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3966825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3966906Z layer_outputs = layer_module( 2025-10-10T01:00:00.3967196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3967290Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3967578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.3967691Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.3967981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3968093Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3968116Z 2025-10-10T01:00:00.3968224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3968430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3968497Z res = mod(**inputs) 2025-10-10T01:00:00.3968784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3968857Z outputs = self.mobilebert( 2025-10-10T01:00:00.3969156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3969231Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3969511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3969590Z layer_outputs = layer_module( 2025-10-10T01:00:00.3969922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3970041Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3970325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3970450Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3970740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.3970828Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3970831Z 2025-10-10T01:00:00.3970944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3971144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3971217Z res = mod(**inputs) 2025-10-10T01:00:00.3971504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3971578Z outputs = self.mobilebert( 2025-10-10T01:00:00.3971869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3971944Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3972237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3972311Z layer_outputs = layer_module( 2025-10-10T01:00:00.3972595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.3972699Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.3972986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.3973125Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.3973409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.3973540Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3973822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3973919Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3973922Z 2025-10-10T01:00:00.3974035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3974237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3974310Z res = mod(**inputs) 2025-10-10T01:00:00.3974593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3974683Z outputs = self.mobilebert( 2025-10-10T01:00:00.3974977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3975050Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3975362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3975438Z layer_outputs = layer_module( 2025-10-10T01:00:00.3975767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3975900Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3976199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.3976332Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.3976336Z 2025-10-10T01:00:00.3976448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3976663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3976734Z res = mod(**inputs) 2025-10-10T01:00:00.3977031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3977114Z outputs = self.mobilebert( 2025-10-10T01:00:00.3977413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3977500Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3977796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3977882Z layer_outputs = layer_module( 2025-10-10T01:00:00.3978181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.3978309Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.3978608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.3978721Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.3978725Z 2025-10-10T01:00:00.3978839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3979047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3979118Z res = mod(**inputs) 2025-10-10T01:00:00.3979431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3979511Z outputs = self.mobilebert( 2025-10-10T01:00:00.3979815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3979894Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3980198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3980275Z layer_outputs = layer_module( 2025-10-10T01:00:00.3980582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3980758Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3981068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.3981176Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.3981198Z 2025-10-10T01:00:00.3981346Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3981564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3981634Z res = mod(**inputs) 2025-10-10T01:00:00.3981941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3982024Z outputs = self.mobilebert( 2025-10-10T01:00:00.3982348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3982435Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3982730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3982805Z layer_outputs = layer_module( 2025-10-10T01:00:00.3983126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3983310Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3983620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.3983752Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.3984067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3984168Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3984172Z 2025-10-10T01:00:00.3984283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3984500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3984572Z res = mod(**inputs) 2025-10-10T01:00:00.3984878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3984955Z outputs = self.mobilebert( 2025-10-10T01:00:00.3985253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3985339Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3985637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3985723Z layer_outputs = layer_module( 2025-10-10T01:00:00.3986039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3986216Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3986608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3986750Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3987063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.3987157Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.3987162Z 2025-10-10T01:00:00.3987283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3987514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3987584Z res = mod(**inputs) 2025-10-10T01:00:00.3987898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3987976Z outputs = self.mobilebert( 2025-10-10T01:00:00.3988306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3988387Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3988696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3988774Z layer_outputs = layer_module( 2025-10-10T01:00:00.3989073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.3989264Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.3989563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.3989702Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.3990025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.3990183Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.3990481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.3990592Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.3990595Z 2025-10-10T01:00:00.3990702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3990892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3990962Z res = mod(**inputs) 2025-10-10T01:00:00.3991229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3991298Z outputs = self.mobilebert( 2025-10-10T01:00:00.3991575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3991647Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3991919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3991989Z layer_outputs = layer_module( 2025-10-10T01:00:00.3992263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3992420Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3992688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.3992804Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.3993072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3993161Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3993164Z 2025-10-10T01:00:00.3993261Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3993449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3993521Z res = mod(**inputs) 2025-10-10T01:00:00.3993795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3993875Z outputs = self.mobilebert( 2025-10-10T01:00:00.3994149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3994236Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3994503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3994593Z layer_outputs = layer_module( 2025-10-10T01:00:00.3994871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.3994957Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.3995235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.3995307Z self_outputs = self.self( 2025-10-10T01:00:00.3995597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.3995677Z self.value(value_tensor) 2025-10-10T01:00:00.3995681Z 2025-10-10T01:00:00.3995783Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3995982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3996087Z res = mod(**inputs) 2025-10-10T01:00:00.3996391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3996468Z outputs = self.mobilebert( 2025-10-10T01:00:00.3996766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3996850Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3997132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3997213Z layer_outputs = layer_module( 2025-10-10T01:00:00.3997493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.3997655Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.3997956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.3998068Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.3998351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.3998436Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.3998440Z 2025-10-10T01:00:00.3998549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.3998747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.3998812Z res = mod(**inputs) 2025-10-10T01:00:00.3999091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.3999164Z outputs = self.mobilebert( 2025-10-10T01:00:00.3999451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.3999524Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.3999796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.3999875Z layer_outputs = layer_module( 2025-10-10T01:00:00.4000152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4000315Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4000588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4000704Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4000993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4001079Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4001354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4001445Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4001448Z 2025-10-10T01:00:00.4001557Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4001765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4001832Z res = mod(**inputs) 2025-10-10T01:00:00.4002116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4002186Z outputs = self.mobilebert( 2025-10-10T01:00:00.4002495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4002570Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4002851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4002922Z layer_outputs = layer_module( 2025-10-10T01:00:00.4003200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4003307Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4003582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4003659Z self_outputs = self.self( 2025-10-10T01:00:00.4003926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4003998Z self.query(query_tensor) 2025-10-10T01:00:00.4004008Z 2025-10-10T01:00:00.4004106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4004295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4004365Z res = mod(**inputs) 2025-10-10T01:00:00.4004641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4004718Z outputs = self.mobilebert( 2025-10-10T01:00:00.4004993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4005069Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4005357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4005434Z layer_outputs = layer_module( 2025-10-10T01:00:00.4005725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4005811Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4006100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4006182Z self_outputs = self.self( 2025-10-10T01:00:00.4006487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4006566Z self.key(key_tensor) 2025-10-10T01:00:00.4006569Z 2025-10-10T01:00:00.4006658Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4006753Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4006866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4007102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4007177Z res = mod(**inputs) 2025-10-10T01:00:00.4007457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4007537Z outputs = self.mobilebert( 2025-10-10T01:00:00.4007817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4007891Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4008191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4008266Z layer_outputs = layer_module( 2025-10-10T01:00:00.4008565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4008686Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4008973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4009109Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4009393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4009488Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4009492Z 2025-10-10T01:00:00.4009598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4009805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4009872Z res = mod(**inputs) 2025-10-10T01:00:00.4010158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4010243Z outputs = self.mobilebert( 2025-10-10T01:00:00.4010525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4010606Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4010889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4010962Z layer_outputs = layer_module( 2025-10-10T01:00:00.4011255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4011342Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4011632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4011757Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4012053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4012181Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4012465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4012567Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4012571Z 2025-10-10T01:00:00.4012679Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4012887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4012954Z res = mod(**inputs) 2025-10-10T01:00:00.4013247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4013339Z outputs = self.mobilebert( 2025-10-10T01:00:00.4013625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4013706Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4013988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4014067Z layer_outputs = layer_module( 2025-10-10T01:00:00.4014376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4014477Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4014765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4014880Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4015184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4015292Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4015296Z 2025-10-10T01:00:00.4015406Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4015602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4015667Z res = mod(**inputs) 2025-10-10T01:00:00.4015954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4016027Z outputs = self.mobilebert( 2025-10-10T01:00:00.4016311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4016384Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4016664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4016746Z layer_outputs = layer_module( 2025-10-10T01:00:00.4017022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4017125Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4017404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4017527Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4017806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4017919Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4017923Z 2025-10-10T01:00:00.4018034Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4018237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4018309Z res = mod(**inputs) 2025-10-10T01:00:00.4018592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4018664Z outputs = self.mobilebert( 2025-10-10T01:00:00.4018949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4019026Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4019311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4019383Z layer_outputs = layer_module( 2025-10-10T01:00:00.4019668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4019791Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4020089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4020235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4020540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4020638Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4020660Z 2025-10-10T01:00:00.4020775Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4020994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4021068Z res = mod(**inputs) 2025-10-10T01:00:00.4021354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4021466Z outputs = self.mobilebert( 2025-10-10T01:00:00.4021762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4021849Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4022158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4022235Z layer_outputs = layer_module( 2025-10-10T01:00:00.4022539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4022641Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4022944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4023081Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4023377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4023515Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4023819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4023928Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4023932Z 2025-10-10T01:00:00.4024044Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4024260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4024329Z res = mod(**inputs) 2025-10-10T01:00:00.4024632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4024722Z outputs = self.mobilebert( 2025-10-10T01:00:00.4025016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4025100Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4025405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4025481Z layer_outputs = layer_module( 2025-10-10T01:00:00.4025797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4025896Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4026204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4026383Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4026735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4026829Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4026834Z 2025-10-10T01:00:00.4026945Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4027172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4027244Z res = mod(**inputs) 2025-10-10T01:00:00.4027582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4027663Z outputs = self.mobilebert( 2025-10-10T01:00:00.4027982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4028069Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4028387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4028492Z layer_outputs = layer_module( 2025-10-10T01:00:00.4028791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4028900Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4029211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4029334Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4029652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4029772Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4029776Z 2025-10-10T01:00:00.4029898Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4030111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4030188Z res = mod(**inputs) 2025-10-10T01:00:00.4030487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4030564Z outputs = self.mobilebert( 2025-10-10T01:00:00.4030869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4030949Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4031255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4031332Z layer_outputs = layer_module( 2025-10-10T01:00:00.4031814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4031932Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4032232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4032376Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4032679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4032781Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4032785Z 2025-10-10T01:00:00.4032898Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4033111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4033191Z res = mod(**inputs) 2025-10-10T01:00:00.4033493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4033629Z outputs = self.mobilebert( 2025-10-10T01:00:00.4033924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4034010Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4034297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4034371Z layer_outputs = layer_module( 2025-10-10T01:00:00.4034676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4034773Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4035060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4035233Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4035519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4035653Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4035934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4036036Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4036040Z 2025-10-10T01:00:00.4036144Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4036343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4036415Z res = mod(**inputs) 2025-10-10T01:00:00.4036699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4036784Z outputs = self.mobilebert( 2025-10-10T01:00:00.4037064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4037142Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4037425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4037497Z layer_outputs = layer_module( 2025-10-10T01:00:00.4037786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4037883Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4038169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4038283Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4038568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4038660Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4038663Z 2025-10-10T01:00:00.4038767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4038974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4039040Z res = mod(**inputs) 2025-10-10T01:00:00.4039330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4039402Z outputs = self.mobilebert( 2025-10-10T01:00:00.4039684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4039764Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4040081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4040162Z layer_outputs = layer_module( 2025-10-10T01:00:00.4040449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4040544Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4040853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4040968Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4041258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4041372Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4041377Z 2025-10-10T01:00:00.4041524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4041724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4041789Z res = mod(**inputs) 2025-10-10T01:00:00.4042077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4042150Z outputs = self.mobilebert( 2025-10-10T01:00:00.4042440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4042515Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4042795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4042874Z layer_outputs = layer_module( 2025-10-10T01:00:00.4043156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4043261Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4043540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4043674Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4043955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4044043Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4044046Z 2025-10-10T01:00:00.4044158Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4044355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4044429Z res = mod(**inputs) 2025-10-10T01:00:00.4044711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4044787Z outputs = self.mobilebert( 2025-10-10T01:00:00.4045073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4045147Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4045433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4045506Z layer_outputs = layer_module( 2025-10-10T01:00:00.4045796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4045890Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4046173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4046331Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4046612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4046744Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4047027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4047127Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4047146Z 2025-10-10T01:00:00.4047252Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4047450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4047523Z res = mod(**inputs) 2025-10-10T01:00:00.4047822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4047919Z outputs = self.mobilebert( 2025-10-10T01:00:00.4048202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4048276Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4048565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4048636Z layer_outputs = layer_module( 2025-10-10T01:00:00.4048935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4049058Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4049347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4049437Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4049441Z 2025-10-10T01:00:00.4049545Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4049752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4049817Z res = mod(**inputs) 2025-10-10T01:00:00.4050108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4050179Z outputs = self.mobilebert( 2025-10-10T01:00:00.4050462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4050544Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4050826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4050906Z layer_outputs = layer_module( 2025-10-10T01:00:00.4051200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4051324Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4051600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4051710Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4051714Z 2025-10-10T01:00:00.4051828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4052027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4052101Z res = mod(**inputs) 2025-10-10T01:00:00.4052384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4052477Z outputs = self.mobilebert( 2025-10-10T01:00:00.4052774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4052849Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4053138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4053211Z layer_outputs = layer_module( 2025-10-10T01:00:00.4053517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4053688Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4053962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4054066Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4054086Z 2025-10-10T01:00:00.4054205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4054408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4054473Z res = mod(**inputs) 2025-10-10T01:00:00.4054754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4054832Z outputs = self.mobilebert( 2025-10-10T01:00:00.4055111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4055190Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4055470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4055550Z layer_outputs = layer_module( 2025-10-10T01:00:00.4055838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4055999Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4056289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4056415Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4056706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4056800Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4056804Z 2025-10-10T01:00:00.4056915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4057115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4057184Z res = mod(**inputs) 2025-10-10T01:00:00.4057485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4057555Z outputs = self.mobilebert( 2025-10-10T01:00:00.4057837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4057910Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4058194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4058273Z layer_outputs = layer_module( 2025-10-10T01:00:00.4058560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4058724Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4059023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4059154Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4059434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4059520Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4059524Z 2025-10-10T01:00:00.4059635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4059874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4059953Z res = mod(**inputs) 2025-10-10T01:00:00.4060259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4060337Z outputs = self.mobilebert( 2025-10-10T01:00:00.4060673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4060778Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4061082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4061159Z layer_outputs = layer_module( 2025-10-10T01:00:00.4061475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4061643Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4061941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4062080Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4062379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4062520Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4062820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4062920Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4062930Z 2025-10-10T01:00:00.4063042Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4063253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4063331Z res = mod(**inputs) 2025-10-10T01:00:00.4063639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4063722Z outputs = self.mobilebert( 2025-10-10T01:00:00.4064033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4064116Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4064424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4064501Z layer_outputs = layer_module( 2025-10-10T01:00:00.4064805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4064969Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4065261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4065388Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4065701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4065832Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4065836Z 2025-10-10T01:00:00.4065946Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4066161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4066231Z res = mod(**inputs) 2025-10-10T01:00:00.4066611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4066723Z outputs = self.mobilebert( 2025-10-10T01:00:00.4067022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4067110Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4067431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4067529Z layer_outputs = layer_module( 2025-10-10T01:00:00.4067834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4067930Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4068298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4068377Z self_outputs = self.self( 2025-10-10T01:00:00.4068693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4068773Z self.value(value_tensor) 2025-10-10T01:00:00.4068776Z 2025-10-10T01:00:00.4068887Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4069104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4069180Z res = mod(**inputs) 2025-10-10T01:00:00.4069488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4069564Z outputs = self.mobilebert( 2025-10-10T01:00:00.4069873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4069957Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4070255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4070340Z layer_outputs = layer_module( 2025-10-10T01:00:00.4070645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4070820Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4071125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4071248Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4071558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4071646Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4071650Z 2025-10-10T01:00:00.4071769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4071978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4072047Z res = mod(**inputs) 2025-10-10T01:00:00.4072353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4072452Z outputs = self.mobilebert( 2025-10-10T01:00:00.4072761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4072840Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4073141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4073219Z layer_outputs = layer_module( 2025-10-10T01:00:00.4073562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4073743Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4074045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4074172Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4074503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4074609Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4074908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4075010Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4075013Z 2025-10-10T01:00:00.4075131Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4075344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4075427Z res = mod(**inputs) 2025-10-10T01:00:00.4075727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4075803Z outputs = self.mobilebert( 2025-10-10T01:00:00.4076117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4076197Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4076501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4076577Z layer_outputs = layer_module( 2025-10-10T01:00:00.4076885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4076978Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4077288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4077375Z self_outputs = self.self( 2025-10-10T01:00:00.4077684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4077778Z self.query(query_tensor) 2025-10-10T01:00:00.4077781Z 2025-10-10T01:00:00.4077886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4078086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4078160Z res = mod(**inputs) 2025-10-10T01:00:00.4078444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4078523Z outputs = self.mobilebert( 2025-10-10T01:00:00.4078807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4078882Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4079172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4079266Z layer_outputs = layer_module( 2025-10-10T01:00:00.4079555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4079643Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4079929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4080001Z self_outputs = self.self( 2025-10-10T01:00:00.4080295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4080375Z self.key(key_tensor) 2025-10-10T01:00:00.4080378Z 2025-10-10T01:00:00.4080463Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4080550Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4080653Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4080882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4080958Z res = mod(**inputs) 2025-10-10T01:00:00.4081238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4081317Z outputs = self.mobilebert( 2025-10-10T01:00:00.4081594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4081675Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4081954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4082028Z layer_outputs = layer_module( 2025-10-10T01:00:00.4082313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4082403Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4082687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4082812Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4083089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4083183Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4083187Z 2025-10-10T01:00:00.4083292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4083498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4083564Z res = mod(**inputs) 2025-10-10T01:00:00.4083847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4083923Z outputs = self.mobilebert( 2025-10-10T01:00:00.4084203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4084283Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4084562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4084644Z layer_outputs = layer_module( 2025-10-10T01:00:00.4084939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4085029Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4085330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4085465Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4085799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4085938Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4086252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4086355Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4086359Z 2025-10-10T01:00:00.4086472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4086714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4086799Z res = mod(**inputs) 2025-10-10T01:00:00.4087101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4087181Z outputs = self.mobilebert( 2025-10-10T01:00:00.4087520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4087608Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4087905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4087991Z layer_outputs = layer_module( 2025-10-10T01:00:00.4088293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4088402Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4088701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4088827Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4089148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4089244Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4089248Z 2025-10-10T01:00:00.4089368Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4089586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4089659Z res = mod(**inputs) 2025-10-10T01:00:00.4089972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4090050Z outputs = self.mobilebert( 2025-10-10T01:00:00.4090362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4090443Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4090756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4090839Z layer_outputs = layer_module( 2025-10-10T01:00:00.4091146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4091259Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4091564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4091697Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4092005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4092130Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4092143Z 2025-10-10T01:00:00.4092255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4092492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4092571Z res = mod(**inputs) 2025-10-10T01:00:00.4092878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4092964Z outputs = self.mobilebert( 2025-10-10T01:00:00.4093270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4093369Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4093686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4093775Z layer_outputs = layer_module( 2025-10-10T01:00:00.4094078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4094216Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4094522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4094668Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4094979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4095080Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4095084Z 2025-10-10T01:00:00.4095200Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4095424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4095495Z res = mod(**inputs) 2025-10-10T01:00:00.4095801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4095892Z outputs = self.mobilebert( 2025-10-10T01:00:00.4096203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4096289Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4096597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4096675Z layer_outputs = layer_module( 2025-10-10T01:00:00.4096990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4097094Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4097405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4097545Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4097862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4097996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4098304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4098415Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4098419Z 2025-10-10T01:00:00.4098533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4098757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4098830Z res = mod(**inputs) 2025-10-10T01:00:00.4099147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4099247Z outputs = self.mobilebert( 2025-10-10T01:00:00.4099554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4099642Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4099949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4100036Z layer_outputs = layer_module( 2025-10-10T01:00:00.4100362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4100468Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4100778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4100900Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4101246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4101342Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4101346Z 2025-10-10T01:00:00.4101465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4101682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4101754Z res = mod(**inputs) 2025-10-10T01:00:00.4102069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4102147Z outputs = self.mobilebert( 2025-10-10T01:00:00.4102461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4102540Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4102852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4102940Z layer_outputs = layer_module( 2025-10-10T01:00:00.4103249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4103360Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4103678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4103810Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4104117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4104242Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4104246Z 2025-10-10T01:00:00.4104368Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4104591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4104672Z res = mod(**inputs) 2025-10-10T01:00:00.4104989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4105070Z outputs = self.mobilebert( 2025-10-10T01:00:00.4105394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4105477Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4105796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4105876Z layer_outputs = layer_module( 2025-10-10T01:00:00.4106199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4106392Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4106720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4106870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4107189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4107291Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4107317Z 2025-10-10T01:00:00.4107434Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4107653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4107736Z res = mod(**inputs) 2025-10-10T01:00:00.4108072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4108179Z outputs = self.mobilebert( 2025-10-10T01:00:00.4108489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4108578Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4108898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4108979Z layer_outputs = layer_module( 2025-10-10T01:00:00.4109299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4109403Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4109731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4109876Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4110185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4110330Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4110649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4110758Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4110762Z 2025-10-10T01:00:00.4110877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4111104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4111176Z res = mod(**inputs) 2025-10-10T01:00:00.4111496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4111588Z outputs = self.mobilebert( 2025-10-10T01:00:00.4111898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4111986Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4112305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4112385Z layer_outputs = layer_module( 2025-10-10T01:00:00.4112711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4112814Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4113129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4113254Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4113589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4113681Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4113685Z 2025-10-10T01:00:00.4113809Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4114025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4114096Z res = mod(**inputs) 2025-10-10T01:00:00.4114427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4114505Z outputs = self.mobilebert( 2025-10-10T01:00:00.4114806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4114892Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4115223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4115312Z layer_outputs = layer_module( 2025-10-10T01:00:00.4115623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4115734Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4116045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4116170Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4116485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4116608Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4116614Z 2025-10-10T01:00:00.4116736Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4116954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4117033Z res = mod(**inputs) 2025-10-10T01:00:00.4117344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4117423Z outputs = self.mobilebert( 2025-10-10T01:00:00.4117739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4117830Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4118135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4118212Z layer_outputs = layer_module( 2025-10-10T01:00:00.4118513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4118624Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4118930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4119070Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4119368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4119465Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4119469Z 2025-10-10T01:00:00.4119579Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4119790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4119869Z res = mod(**inputs) 2025-10-10T01:00:00.4120171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4120276Z outputs = self.mobilebert( 2025-10-10T01:00:00.4120580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4120658Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4120971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4121048Z layer_outputs = layer_module( 2025-10-10T01:00:00.4121378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4121480Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4121786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4121952Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4122249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4122388Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4122685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4122794Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4122800Z 2025-10-10T01:00:00.4122912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4123123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4123200Z res = mod(**inputs) 2025-10-10T01:00:00.4123499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4123586Z outputs = self.mobilebert( 2025-10-10T01:00:00.4123882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4123968Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4124266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4124343Z layer_outputs = layer_module( 2025-10-10T01:00:00.4124650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4124782Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4125088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4125185Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4125189Z 2025-10-10T01:00:00.4125300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4125520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4125590Z res = mod(**inputs) 2025-10-10T01:00:00.4125892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4125969Z outputs = self.mobilebert( 2025-10-10T01:00:00.4126275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4126353Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4126649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4126757Z layer_outputs = layer_module( 2025-10-10T01:00:00.4127059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4127198Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4127505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4127629Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4127641Z 2025-10-10T01:00:00.4127778Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4127991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4128070Z res = mod(**inputs) 2025-10-10T01:00:00.4128368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4128451Z outputs = self.mobilebert( 2025-10-10T01:00:00.4128786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4128865Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4129176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4129255Z layer_outputs = layer_module( 2025-10-10T01:00:00.4129572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4129758Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4130056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4130169Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4130176Z 2025-10-10T01:00:00.4130287Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4130507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4130577Z res = mod(**inputs) 2025-10-10T01:00:00.4130884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4130962Z outputs = self.mobilebert( 2025-10-10T01:00:00.4131264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4131351Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4131773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4131863Z layer_outputs = layer_module( 2025-10-10T01:00:00.4132168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4132339Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4132650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4132783Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4133097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4133200Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4133204Z 2025-10-10T01:00:00.4133323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4133537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4133650Z res = mod(**inputs) 2025-10-10T01:00:00.4133956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4134031Z outputs = self.mobilebert( 2025-10-10T01:00:00.4134333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4134408Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4134715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4134815Z layer_outputs = layer_module( 2025-10-10T01:00:00.4135117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4135294Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4135628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4135796Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4136103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4136198Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4136210Z 2025-10-10T01:00:00.4136324Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4136543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4136619Z res = mod(**inputs) 2025-10-10T01:00:00.4136926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4137011Z outputs = self.mobilebert( 2025-10-10T01:00:00.4137322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4137403Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4137714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4137793Z layer_outputs = layer_module( 2025-10-10T01:00:00.4138105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4138275Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4138582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4138729Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4139036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4139179Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4139484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4139597Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4139600Z 2025-10-10T01:00:00.4139713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4139933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4140014Z res = mod(**inputs) 2025-10-10T01:00:00.4140322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4140407Z outputs = self.mobilebert( 2025-10-10T01:00:00.4140718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4140828Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4141143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4141222Z layer_outputs = layer_module( 2025-10-10T01:00:00.4141533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4141726Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4142046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4142169Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4142497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4142621Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4142625Z 2025-10-10T01:00:00.4142743Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4142969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4143043Z res = mod(**inputs) 2025-10-10T01:00:00.4143361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4143442Z outputs = self.mobilebert( 2025-10-10T01:00:00.4143749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4143838Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4144155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4144244Z layer_outputs = layer_module( 2025-10-10T01:00:00.4144550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4144649Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4144971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4145050Z self_outputs = self.self( 2025-10-10T01:00:00.4145369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4145449Z self.value(value_tensor) 2025-10-10T01:00:00.4145452Z 2025-10-10T01:00:00.4145574Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4145788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4145867Z res = mod(**inputs) 2025-10-10T01:00:00.4146184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4146262Z outputs = self.mobilebert( 2025-10-10T01:00:00.4146636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4146720Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4147041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4147129Z layer_outputs = layer_module( 2025-10-10T01:00:00.4147442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4147625Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4147956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4148088Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4148393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4148485Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4148489Z 2025-10-10T01:00:00.4148611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4148846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4148939Z res = mod(**inputs) 2025-10-10T01:00:00.4149222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4149296Z outputs = self.mobilebert( 2025-10-10T01:00:00.4149624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4149705Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4150013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4150092Z layer_outputs = layer_module( 2025-10-10T01:00:00.4150411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4150580Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4150897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4151024Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4151332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4151431Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4151733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4151832Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4151844Z 2025-10-10T01:00:00.4151953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4152165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4152243Z res = mod(**inputs) 2025-10-10T01:00:00.4152544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4152625Z outputs = self.mobilebert( 2025-10-10T01:00:00.4152931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4153009Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4153320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4153396Z layer_outputs = layer_module( 2025-10-10T01:00:00.4153711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4185721Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4186272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4186443Z self_outputs = self.self( 2025-10-10T01:00:00.4186792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4187015Z self.query(query_tensor) 2025-10-10T01:00:00.4187024Z 2025-10-10T01:00:00.4187162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4187408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4187489Z res = mod(**inputs) 2025-10-10T01:00:00.4187812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4187920Z outputs = self.mobilebert( 2025-10-10T01:00:00.4188285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4188384Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4188688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4188861Z layer_outputs = layer_module( 2025-10-10T01:00:00.4189163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4189265Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4189572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4189654Z self_outputs = self.self( 2025-10-10T01:00:00.4189963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4190042Z self.key(key_tensor) 2025-10-10T01:00:00.4190046Z 2025-10-10T01:00:00.4190142Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4190241Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4190362Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4190606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4190682Z res = mod(**inputs) 2025-10-10T01:00:00.4190992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4191086Z outputs = self.mobilebert( 2025-10-10T01:00:00.4191396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4191488Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4191789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4191878Z layer_outputs = layer_module( 2025-10-10T01:00:00.4192171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4192273Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4192579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4192716Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4193019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4193113Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4193118Z 2025-10-10T01:00:00.4193235Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4193459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4193529Z res = mod(**inputs) 2025-10-10T01:00:00.4193837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4193944Z outputs = self.mobilebert( 2025-10-10T01:00:00.4194249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4194331Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4194629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4194717Z layer_outputs = layer_module( 2025-10-10T01:00:00.4195036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4195137Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4195436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4195571Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4195927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4196071Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4196380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4196489Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4196493Z 2025-10-10T01:00:00.4196618Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4196833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4196906Z res = mod(**inputs) 2025-10-10T01:00:00.4197210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4197289Z outputs = self.mobilebert( 2025-10-10T01:00:00.4197597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4197681Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4197979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4198069Z layer_outputs = layer_module( 2025-10-10T01:00:00.4198362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4198476Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4198777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4198915Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4199217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4199318Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4199322Z 2025-10-10T01:00:00.4199444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4199663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4199744Z res = mod(**inputs) 2025-10-10T01:00:00.4200055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4200131Z outputs = self.mobilebert( 2025-10-10T01:00:00.4200431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4200510Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4200818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4200914Z layer_outputs = layer_module( 2025-10-10T01:00:00.4201220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4201324Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4201621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4201773Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4202078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4202212Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4202216Z 2025-10-10T01:00:00.4202330Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4202621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4202701Z res = mod(**inputs) 2025-10-10T01:00:00.4203005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4203089Z outputs = self.mobilebert( 2025-10-10T01:00:00.4203395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4203484Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4203791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4203870Z layer_outputs = layer_module( 2025-10-10T01:00:00.4204186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4204297Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4204609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4204755Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4205064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4205169Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4205174Z 2025-10-10T01:00:00.4205292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4205518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4205592Z res = mod(**inputs) 2025-10-10T01:00:00.4205904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4205991Z outputs = self.mobilebert( 2025-10-10T01:00:00.4206299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4206392Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4206700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4206789Z layer_outputs = layer_module( 2025-10-10T01:00:00.4207108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4207213Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4207528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4207665Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4208005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4208144Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4208460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4208567Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4208571Z 2025-10-10T01:00:00.4208702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4208929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4209002Z res = mod(**inputs) 2025-10-10T01:00:00.4209322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4209437Z outputs = self.mobilebert( 2025-10-10T01:00:00.4209751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4209834Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4210152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4210242Z layer_outputs = layer_module( 2025-10-10T01:00:00.4210560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4210673Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4210988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4211116Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4211442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4211537Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4211541Z 2025-10-10T01:00:00.4211663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4211881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4211963Z res = mod(**inputs) 2025-10-10T01:00:00.4212273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4212354Z outputs = self.mobilebert( 2025-10-10T01:00:00.4212675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4212755Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4213083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4213166Z layer_outputs = layer_module( 2025-10-10T01:00:00.4213473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4213588Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4213905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4214050Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4214360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4214492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4214496Z 2025-10-10T01:00:00.4214609Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4214853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4214935Z res = mod(**inputs) 2025-10-10T01:00:00.4215234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4215322Z outputs = self.mobilebert( 2025-10-10T01:00:00.4215622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4215718Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4216025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4216103Z layer_outputs = layer_module( 2025-10-10T01:00:00.4216414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4216551Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4216869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4217011Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4217322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4217428Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4217434Z 2025-10-10T01:00:00.4217551Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4217777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4217851Z res = mod(**inputs) 2025-10-10T01:00:00.4218164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4218255Z outputs = self.mobilebert( 2025-10-10T01:00:00.4218553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4218639Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4218940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4219025Z layer_outputs = layer_module( 2025-10-10T01:00:00.4219323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4219425Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4219729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4219871Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4220177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4220309Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4220608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4220720Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4220724Z 2025-10-10T01:00:00.4220839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4221057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4221128Z res = mod(**inputs) 2025-10-10T01:00:00.4221437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4221537Z outputs = self.mobilebert( 2025-10-10T01:00:00.4221842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4221929Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4222234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4222320Z layer_outputs = layer_module( 2025-10-10T01:00:00.4222648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4222750Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4223053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4223177Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4223513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4223607Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4223611Z 2025-10-10T01:00:00.4223728Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4223938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4224008Z res = mod(**inputs) 2025-10-10T01:00:00.4224317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4224395Z outputs = self.mobilebert( 2025-10-10T01:00:00.4224699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4224779Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4225080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4225170Z layer_outputs = layer_module( 2025-10-10T01:00:00.4225470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4225582Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4225887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4226021Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4226422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4226562Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4226570Z 2025-10-10T01:00:00.4226693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4226912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4226988Z res = mod(**inputs) 2025-10-10T01:00:00.4227294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4227382Z outputs = self.mobilebert( 2025-10-10T01:00:00.4227691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4227769Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4228068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4228141Z layer_outputs = layer_module( 2025-10-10T01:00:00.4228456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4228581Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4228874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4229013Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4229306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4229420Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4229424Z 2025-10-10T01:00:00.4229536Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4229754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4229822Z res = mod(**inputs) 2025-10-10T01:00:00.4230145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4230241Z outputs = self.mobilebert( 2025-10-10T01:00:00.4230534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4230613Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4230916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4230990Z layer_outputs = layer_module( 2025-10-10T01:00:00.4231303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4231397Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4231987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4232134Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4232445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4232579Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4232877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4232991Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4232997Z 2025-10-10T01:00:00.4233107Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4233324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4233396Z res = mod(**inputs) 2025-10-10T01:00:00.4233692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4233774Z outputs = self.mobilebert( 2025-10-10T01:00:00.4234068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4234156Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4234462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4234543Z layer_outputs = layer_module( 2025-10-10T01:00:00.4234828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4234954Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4235243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4235409Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4235412Z 2025-10-10T01:00:00.4235528Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4235739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4235812Z res = mod(**inputs) 2025-10-10T01:00:00.4236132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4236210Z outputs = self.mobilebert( 2025-10-10T01:00:00.4236546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4236626Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4236931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4237007Z layer_outputs = layer_module( 2025-10-10T01:00:00.4237369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4237511Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4237808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4237939Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4237943Z 2025-10-10T01:00:00.4238055Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4238268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4238349Z res = mod(**inputs) 2025-10-10T01:00:00.4238646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4238734Z outputs = self.mobilebert( 2025-10-10T01:00:00.4239035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4239121Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4239420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4239498Z layer_outputs = layer_module( 2025-10-10T01:00:00.4239803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4239967Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4240256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4240353Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4240360Z 2025-10-10T01:00:00.4240466Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4240665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4240732Z res = mod(**inputs) 2025-10-10T01:00:00.4241022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4241096Z outputs = self.mobilebert( 2025-10-10T01:00:00.4241389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4241463Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4241746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4241828Z layer_outputs = layer_module( 2025-10-10T01:00:00.4242110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4242298Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4242579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4242714Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4243012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4243111Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4243114Z 2025-10-10T01:00:00.4243226Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4243425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4243502Z res = mod(**inputs) 2025-10-10T01:00:00.4243817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4243891Z outputs = self.mobilebert( 2025-10-10T01:00:00.4244179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4244252Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4244541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4244616Z layer_outputs = layer_module( 2025-10-10T01:00:00.4244913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4245071Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4245354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4245495Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4245776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4245872Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4245876Z 2025-10-10T01:00:00.4245979Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4246182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4246260Z res = mod(**inputs) 2025-10-10T01:00:00.4246555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4246640Z outputs = self.mobilebert( 2025-10-10T01:00:00.4246939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4247029Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4247328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4247406Z layer_outputs = layer_module( 2025-10-10T01:00:00.4247713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4247882Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4248189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4248316Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4248598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4248750Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4249032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4249137Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4249141Z 2025-10-10T01:00:00.4249251Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4249489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4249562Z res = mod(**inputs) 2025-10-10T01:00:00.4249860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4249950Z outputs = self.mobilebert( 2025-10-10T01:00:00.4250265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4250374Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4250672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4250750Z layer_outputs = layer_module( 2025-10-10T01:00:00.4251058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4251237Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4251547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4251670Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4251990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4252083Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4252087Z 2025-10-10T01:00:00.4252197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4252416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4252482Z res = mod(**inputs) 2025-10-10T01:00:00.4252786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4252861Z outputs = self.mobilebert( 2025-10-10T01:00:00.4253163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4253241Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4253545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4253635Z layer_outputs = layer_module( 2025-10-10T01:00:00.4253932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4254035Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4254344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4254424Z self_outputs = self.self( 2025-10-10T01:00:00.4254743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4254821Z self.value(value_tensor) 2025-10-10T01:00:00.4254824Z 2025-10-10T01:00:00.4254943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4255155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4255253Z res = mod(**inputs) 2025-10-10T01:00:00.4255552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4255630Z outputs = self.mobilebert( 2025-10-10T01:00:00.4255939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4256019Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4256347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4256424Z layer_outputs = layer_module( 2025-10-10T01:00:00.4256717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4256895Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4257228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4257355Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4257650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4257748Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4257753Z 2025-10-10T01:00:00.4257862Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4258075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4258154Z res = mod(**inputs) 2025-10-10T01:00:00.4258451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4258535Z outputs = self.mobilebert( 2025-10-10T01:00:00.4258835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4258914Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4259220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4259297Z layer_outputs = layer_module( 2025-10-10T01:00:00.4259605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4259774Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4260083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4260202Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4260505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4260606Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4260933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4261042Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4261046Z 2025-10-10T01:00:00.4261154Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4261364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4261442Z res = mod(**inputs) 2025-10-10T01:00:00.4261739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4261825Z outputs = self.mobilebert( 2025-10-10T01:00:00.4262156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4262236Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4262537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4262615Z layer_outputs = layer_module( 2025-10-10T01:00:00.4262920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4263028Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4263334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4263423Z self_outputs = self.self( 2025-10-10T01:00:00.4263751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4263857Z self.query(query_tensor) 2025-10-10T01:00:00.4263862Z 2025-10-10T01:00:00.4263976Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4264194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4264274Z res = mod(**inputs) 2025-10-10T01:00:00.4264585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4264672Z outputs = self.mobilebert( 2025-10-10T01:00:00.4264983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4265070Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4265383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4265465Z layer_outputs = layer_module( 2025-10-10T01:00:00.4265781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4265878Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4266193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4266272Z self_outputs = self.self( 2025-10-10T01:00:00.4266661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4266753Z self.key(key_tensor) 2025-10-10T01:00:00.4266757Z 2025-10-10T01:00:00.4266853Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4266954Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4267070Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4267289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4267374Z res = mod(**inputs) 2025-10-10T01:00:00.4267682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4267767Z outputs = self.mobilebert( 2025-10-10T01:00:00.4268074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4268159Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4268473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4268553Z layer_outputs = layer_module( 2025-10-10T01:00:00.4268871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4268997Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4269307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4269446Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4269763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4269865Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4269869Z 2025-10-10T01:00:00.4270000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4270226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4270297Z res = mod(**inputs) 2025-10-10T01:00:00.4270621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4270742Z outputs = self.mobilebert( 2025-10-10T01:00:00.4271048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4271137Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4271448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4271530Z layer_outputs = layer_module( 2025-10-10T01:00:00.4271847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4271940Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4272256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4272391Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4272707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4272846Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4273153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4273254Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4273258Z 2025-10-10T01:00:00.4273369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4273592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4273664Z res = mod(**inputs) 2025-10-10T01:00:00.4273983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4274061Z outputs = self.mobilebert( 2025-10-10T01:00:00.4274372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4274458Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4274761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4274846Z layer_outputs = layer_module( 2025-10-10T01:00:00.4275151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4275276Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4275556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4275670Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4275963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4276076Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4276080Z 2025-10-10T01:00:00.4276197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4276404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4276482Z res = mod(**inputs) 2025-10-10T01:00:00.4276788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4276861Z outputs = self.mobilebert( 2025-10-10T01:00:00.4277145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4277218Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4277538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4277634Z layer_outputs = layer_module( 2025-10-10T01:00:00.4277939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4278041Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4278322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4278443Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4278733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4278860Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4278864Z 2025-10-10T01:00:00.4278973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4279187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4279265Z res = mod(**inputs) 2025-10-10T01:00:00.4279554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4279631Z outputs = self.mobilebert( 2025-10-10T01:00:00.4279908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4279978Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4280264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4280337Z layer_outputs = layer_module( 2025-10-10T01:00:00.4280626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4280726Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4281016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4281145Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4281440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4281539Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4281543Z 2025-10-10T01:00:00.4281655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4281874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4281942Z res = mod(**inputs) 2025-10-10T01:00:00.4282240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4282346Z outputs = self.mobilebert( 2025-10-10T01:00:00.4282640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4282721Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4283002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4283079Z layer_outputs = layer_module( 2025-10-10T01:00:00.4283392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4283495Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4283809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4283935Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4284263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4284390Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4284670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4284774Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4284777Z 2025-10-10T01:00:00.4284881Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4285088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4285153Z res = mod(**inputs) 2025-10-10T01:00:00.4285438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4285513Z outputs = self.mobilebert( 2025-10-10T01:00:00.4285794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4285874Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4286152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4286232Z layer_outputs = layer_module( 2025-10-10T01:00:00.4286513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4286609Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4286891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4287001Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4287289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4287376Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4287380Z 2025-10-10T01:00:00.4287489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4287688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4287756Z res = mod(**inputs) 2025-10-10T01:00:00.4288047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4288120Z outputs = self.mobilebert( 2025-10-10T01:00:00.4288408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4288481Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4288772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4288876Z layer_outputs = layer_module( 2025-10-10T01:00:00.4289173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4289282Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4289578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4289729Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4290027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4290148Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4290151Z 2025-10-10T01:00:00.4290270Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4290513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4290594Z res = mod(**inputs) 2025-10-10T01:00:00.4290894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4290967Z outputs = self.mobilebert( 2025-10-10T01:00:00.4291271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4291350Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4291660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4291735Z layer_outputs = layer_module( 2025-10-10T01:00:00.4292045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4292149Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4292447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4292586Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4292887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4292981Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4292987Z 2025-10-10T01:00:00.4293097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4293314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4293383Z res = mod(**inputs) 2025-10-10T01:00:00.4293686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4293772Z outputs = self.mobilebert( 2025-10-10T01:00:00.4294076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4294159Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4294461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4294536Z layer_outputs = layer_module( 2025-10-10T01:00:00.4294847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4294948Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4295258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4295413Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4295720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4295850Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4296148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4296254Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4296258Z 2025-10-10T01:00:00.4296385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4296605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4296676Z res = mod(**inputs) 2025-10-10T01:00:00.4296974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4297098Z outputs = self.mobilebert( 2025-10-10T01:00:00.4297402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4297488Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4297788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4297873Z layer_outputs = layer_module( 2025-10-10T01:00:00.4298173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4298273Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4298576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4298695Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4299003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4299094Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4299098Z 2025-10-10T01:00:00.4299205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4299421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4299490Z res = mod(**inputs) 2025-10-10T01:00:00.4299798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4299876Z outputs = self.mobilebert( 2025-10-10T01:00:00.4300184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4300261Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4300565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4300649Z layer_outputs = layer_module( 2025-10-10T01:00:00.4300946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4301053Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4301353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4301476Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4301783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4301903Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4301907Z 2025-10-10T01:00:00.4302048Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4302260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4302335Z res = mod(**inputs) 2025-10-10T01:00:00.4302631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4302707Z outputs = self.mobilebert( 2025-10-10T01:00:00.4303017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4303118Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4303422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4303499Z layer_outputs = layer_module( 2025-10-10T01:00:00.4303812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4303938Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4304237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4304377Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4304675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4304772Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4304776Z 2025-10-10T01:00:00.4304887Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4305102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4305183Z res = mod(**inputs) 2025-10-10T01:00:00.4305488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4305578Z outputs = self.mobilebert( 2025-10-10T01:00:00.4305882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4305961Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4306274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4306430Z layer_outputs = layer_module( 2025-10-10T01:00:00.4306756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4306859Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4307171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4307315Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4307620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4307760Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4308074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4308178Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4308184Z 2025-10-10T01:00:00.4308291Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4308505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4308576Z res = mod(**inputs) 2025-10-10T01:00:00.4308874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4308987Z outputs = self.mobilebert( 2025-10-10T01:00:00.4309285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4309372Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4309681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4309759Z layer_outputs = layer_module( 2025-10-10T01:00:00.4310126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4310258Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4310617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4310731Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4310751Z 2025-10-10T01:00:00.4310872Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4311085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4311155Z res = mod(**inputs) 2025-10-10T01:00:00.4311470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4311548Z outputs = self.mobilebert( 2025-10-10T01:00:00.4311934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4312014Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4312325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4312407Z layer_outputs = layer_module( 2025-10-10T01:00:00.4312717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4312853Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4313155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4313287Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4313291Z 2025-10-10T01:00:00.4313405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4313621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4313702Z res = mod(**inputs) 2025-10-10T01:00:00.4314033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4314119Z outputs = self.mobilebert( 2025-10-10T01:00:00.4314427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4314506Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4314817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4314895Z layer_outputs = layer_module( 2025-10-10T01:00:00.4315227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4315400Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4315737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4315842Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4315872Z 2025-10-10T01:00:00.4315990Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4316214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4316285Z res = mod(**inputs) 2025-10-10T01:00:00.4316615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4316693Z outputs = self.mobilebert( 2025-10-10T01:00:00.4317016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4317107Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4317410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4317496Z layer_outputs = layer_module( 2025-10-10T01:00:00.4317842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4318040Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4318369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4318504Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4318819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4318922Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4318926Z 2025-10-10T01:00:00.4319045Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4319261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4319332Z res = mod(**inputs) 2025-10-10T01:00:00.4319652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4319731Z outputs = self.mobilebert( 2025-10-10T01:00:00.4320044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4320122Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4320434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4320513Z layer_outputs = layer_module( 2025-10-10T01:00:00.4320820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4320998Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4321306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4321450Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4321756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4321847Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4321860Z 2025-10-10T01:00:00.4321971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4322190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4322269Z res = mod(**inputs) 2025-10-10T01:00:00.4322585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4322664Z outputs = self.mobilebert( 2025-10-10T01:00:00.4322977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4323088Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4323383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4323466Z layer_outputs = layer_module( 2025-10-10T01:00:00.4323762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4323948Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4324246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4324392Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4324715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4324863Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4325169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4325269Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4325273Z 2025-10-10T01:00:00.4325383Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4325602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4325672Z res = mod(**inputs) 2025-10-10T01:00:00.4325974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4326051Z outputs = self.mobilebert( 2025-10-10T01:00:00.4326348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4326437Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4326733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4326817Z layer_outputs = layer_module( 2025-10-10T01:00:00.4327115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4327296Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4327597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4327716Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4328023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4328116Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4328120Z 2025-10-10T01:00:00.4328238Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4328448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4328519Z res = mod(**inputs) 2025-10-10T01:00:00.4328825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4328905Z outputs = self.mobilebert( 2025-10-10T01:00:00.4329216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4329295Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4329606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4329709Z layer_outputs = layer_module( 2025-10-10T01:00:00.4330030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4330134Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4330433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4330519Z self_outputs = self.self( 2025-10-10T01:00:00.4330833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4330912Z self.value(value_tensor) 2025-10-10T01:00:00.4330923Z 2025-10-10T01:00:00.4331034Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4331242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4331353Z res = mod(**inputs) 2025-10-10T01:00:00.4331850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4331943Z outputs = self.mobilebert( 2025-10-10T01:00:00.4332240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4332320Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4332629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4332706Z layer_outputs = layer_module( 2025-10-10T01:00:00.4333012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4333182Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4333486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4333615Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4333915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4334012Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4334016Z 2025-10-10T01:00:00.4334128Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4334347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4334418Z res = mod(**inputs) 2025-10-10T01:00:00.4334717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4334800Z outputs = self.mobilebert( 2025-10-10T01:00:00.4335103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4335189Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4335487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4335563Z layer_outputs = layer_module( 2025-10-10T01:00:00.4335866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4336037Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4336347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4336464Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4336818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4336912Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4337205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4337312Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4337315Z 2025-10-10T01:00:00.4337425Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4337665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4337737Z res = mod(**inputs) 2025-10-10T01:00:00.4338038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4338115Z outputs = self.mobilebert( 2025-10-10T01:00:00.4338441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4338572Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4338873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4338957Z layer_outputs = layer_module( 2025-10-10T01:00:00.4339257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4339353Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4339664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4339737Z self_outputs = self.self( 2025-10-10T01:00:00.4340028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4340105Z self.query(query_tensor) 2025-10-10T01:00:00.4340108Z 2025-10-10T01:00:00.4340219Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4340417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4340484Z res = mod(**inputs) 2025-10-10T01:00:00.4340809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4340884Z outputs = self.mobilebert( 2025-10-10T01:00:00.4341187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4341264Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4341559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4341652Z layer_outputs = layer_module( 2025-10-10T01:00:00.4341949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4342051Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4342349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4342425Z self_outputs = self.self( 2025-10-10T01:00:00.4342729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4342804Z self.key(key_tensor) 2025-10-10T01:00:00.4342807Z 2025-10-10T01:00:00.4342904Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4342990Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4343108Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4343342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4343414Z res = mod(**inputs) 2025-10-10T01:00:00.4343720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4343795Z outputs = self.mobilebert( 2025-10-10T01:00:00.4344097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4344174Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4344489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4344575Z layer_outputs = layer_module( 2025-10-10T01:00:00.4344872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4345004Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4345299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4345438Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4345733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4345823Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4345826Z 2025-10-10T01:00:00.4345946Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4346153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4346229Z res = mod(**inputs) 2025-10-10T01:00:00.4346578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4346666Z outputs = self.mobilebert( 2025-10-10T01:00:00.4346979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4347059Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4347373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4347451Z layer_outputs = layer_module( 2025-10-10T01:00:00.4347767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4347862Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4348165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4348304Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4348610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4348746Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4349026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4349120Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4349133Z 2025-10-10T01:00:00.4349236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4349435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4349509Z res = mod(**inputs) 2025-10-10T01:00:00.4349788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4349866Z outputs = self.mobilebert( 2025-10-10T01:00:00.4350167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4350240Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4350527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4350598Z layer_outputs = layer_module( 2025-10-10T01:00:00.4350879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4350991Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4351275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4351396Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4351694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4351803Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4351806Z 2025-10-10T01:00:00.4351910Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4352113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4352178Z res = mod(**inputs) 2025-10-10T01:00:00.4352456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4352538Z outputs = self.mobilebert( 2025-10-10T01:00:00.4352818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4352900Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4353181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4353256Z layer_outputs = layer_module( 2025-10-10T01:00:00.4353543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4353641Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4353928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4354041Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4354331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4354445Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4354449Z 2025-10-10T01:00:00.4354552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4354763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4354829Z res = mod(**inputs) 2025-10-10T01:00:00.4355116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4355197Z outputs = self.mobilebert( 2025-10-10T01:00:00.4355470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4355550Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4355823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4355903Z layer_outputs = layer_module( 2025-10-10T01:00:00.4356182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4356306Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4356587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4356715Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4357001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4357087Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4357090Z 2025-10-10T01:00:00.4357216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4357416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4357482Z res = mod(**inputs) 2025-10-10T01:00:00.4357767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4358480Z outputs = self.mobilebert( 2025-10-10T01:00:00.4358773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4358843Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4359116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4359187Z layer_outputs = layer_module( 2025-10-10T01:00:00.4359452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4359551Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4359817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4359947Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4360223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4360347Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4360630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4360723Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4360726Z 2025-10-10T01:00:00.4360837Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4361033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4361105Z res = mod(**inputs) 2025-10-10T01:00:00.4361378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4361450Z outputs = self.mobilebert( 2025-10-10T01:00:00.4361734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4361806Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4362087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4362158Z layer_outputs = layer_module( 2025-10-10T01:00:00.4362438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4362541Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4362823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4362943Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4363242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4363337Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4363341Z 2025-10-10T01:00:00.4363444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4363643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4363715Z res = mod(**inputs) 2025-10-10T01:00:00.4364011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4364092Z outputs = self.mobilebert( 2025-10-10T01:00:00.4364377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4364457Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4364770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4364860Z layer_outputs = layer_module( 2025-10-10T01:00:00.4365143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4365237Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4365521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4365635Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4365915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4366035Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4366039Z 2025-10-10T01:00:00.4366142Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4366351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4366417Z res = mod(**inputs) 2025-10-10T01:00:00.4366699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4366771Z outputs = self.mobilebert( 2025-10-10T01:00:00.4367050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4367132Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4367409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4367489Z layer_outputs = layer_module( 2025-10-10T01:00:00.4367765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4367864Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4368163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4368296Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4368599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4368690Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4368695Z 2025-10-10T01:00:00.4368811Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4369017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4369088Z res = mod(**inputs) 2025-10-10T01:00:00.4369392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4369491Z outputs = self.mobilebert( 2025-10-10T01:00:00.4369793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4369872Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4370171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4370254Z layer_outputs = layer_module( 2025-10-10T01:00:00.4370569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4370682Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4370982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4371142Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4371453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4371585Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4371891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4371992Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4371996Z 2025-10-10T01:00:00.4372114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4372324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4372394Z res = mod(**inputs) 2025-10-10T01:00:00.4372695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4372776Z outputs = self.mobilebert( 2025-10-10T01:00:00.4373078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4373156Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4373461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4373539Z layer_outputs = layer_module( 2025-10-10T01:00:00.4373837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4373948Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4374240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4374367Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4374670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4374759Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4374771Z 2025-10-10T01:00:00.4374880Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4375090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4375167Z res = mod(**inputs) 2025-10-10T01:00:00.4375462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4375544Z outputs = self.mobilebert( 2025-10-10T01:00:00.4375839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4375915Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4376239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4376317Z layer_outputs = layer_module( 2025-10-10T01:00:00.4376620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4376720Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4377017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4377159Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4377455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4377581Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4377585Z 2025-10-10T01:00:00.4377714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4377942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4378012Z res = mod(**inputs) 2025-10-10T01:00:00.4378307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4378391Z outputs = self.mobilebert( 2025-10-10T01:00:00.4378693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4378780Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4379082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4379159Z layer_outputs = layer_module( 2025-10-10T01:00:00.4379470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4379572Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4379881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4380014Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4380324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4380414Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4380418Z 2025-10-10T01:00:00.4380528Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4380748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4380817Z res = mod(**inputs) 2025-10-10T01:00:00.4381131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4381207Z outputs = self.mobilebert( 2025-10-10T01:00:00.4381518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4381598Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4381911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4381997Z layer_outputs = layer_module( 2025-10-10T01:00:00.4382310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4382418Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4382727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4382885Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4383197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4383332Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4383643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4383745Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4383749Z 2025-10-10T01:00:00.4383883Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4384103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4384176Z res = mod(**inputs) 2025-10-10T01:00:00.4384489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4384606Z outputs = self.mobilebert( 2025-10-10T01:00:00.4384919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4385000Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4385303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4385390Z layer_outputs = layer_module( 2025-10-10T01:00:00.4385697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4385840Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4386143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4386244Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4386251Z 2025-10-10T01:00:00.4386441Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4386669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4386753Z res = mod(**inputs) 2025-10-10T01:00:00.4387063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4387152Z outputs = self.mobilebert( 2025-10-10T01:00:00.4387466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4387547Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4387866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4387946Z layer_outputs = layer_module( 2025-10-10T01:00:00.4388269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4388404Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4388721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4388848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4388852Z 2025-10-10T01:00:00.4388969Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4389198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4389273Z res = mod(**inputs) 2025-10-10T01:00:00.4389591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4389670Z outputs = self.mobilebert( 2025-10-10T01:00:00.4390001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4390092Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4390398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4390484Z layer_outputs = layer_module( 2025-10-10T01:00:00.4390789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4390995Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4391300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4391405Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4391411Z 2025-10-10T01:00:00.4391576Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4391793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4391873Z res = mod(**inputs) 2025-10-10T01:00:00.4392179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4392264Z outputs = self.mobilebert( 2025-10-10T01:00:00.4392576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4392654Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4392967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4393047Z layer_outputs = layer_module( 2025-10-10T01:00:00.4393360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4393536Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4393856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4394003Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4394312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4394425Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4394429Z 2025-10-10T01:00:00.4394543Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4394766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4394837Z res = mod(**inputs) 2025-10-10T01:00:00.4395150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4395236Z outputs = self.mobilebert( 2025-10-10T01:00:00.4395540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4395627Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4395930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4396011Z layer_outputs = layer_module( 2025-10-10T01:00:00.4396323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4396493Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4396807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4396966Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4397281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4397377Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4397381Z 2025-10-10T01:00:00.4397495Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4397735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4397808Z res = mod(**inputs) 2025-10-10T01:00:00.4398128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4398207Z outputs = self.mobilebert( 2025-10-10T01:00:00.4398536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4398638Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4398939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4399021Z layer_outputs = layer_module( 2025-10-10T01:00:00.4399324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4399501Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4399816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4399951Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4400264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4400400Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4400713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4400815Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4400819Z 2025-10-10T01:00:00.4400939Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4401155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4401226Z res = mod(**inputs) 2025-10-10T01:00:00.4401546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4401619Z outputs = self.mobilebert( 2025-10-10T01:00:00.4401921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4402000Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4402294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4402377Z layer_outputs = layer_module( 2025-10-10T01:00:00.4402671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4402850Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4403148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4403274Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4403582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4403691Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4403695Z 2025-10-10T01:00:00.4403813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4404043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4404120Z res = mod(**inputs) 2025-10-10T01:00:00.4404430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4404522Z outputs = self.mobilebert( 2025-10-10T01:00:00.4404838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4404917Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4405229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4405346Z layer_outputs = layer_module( 2025-10-10T01:00:00.4405662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4405759Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4406064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4406149Z self_outputs = self.self( 2025-10-10T01:00:00.4406460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4406544Z self.value(value_tensor) 2025-10-10T01:00:00.4406548Z 2025-10-10T01:00:00.4406659Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4406869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4406957Z res = mod(**inputs) 2025-10-10T01:00:00.4407271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4407354Z outputs = self.mobilebert( 2025-10-10T01:00:00.4407666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4407746Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4408067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4408146Z layer_outputs = layer_module( 2025-10-10T01:00:00.4408470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4408639Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4408959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4409078Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4409381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4409476Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4409480Z 2025-10-10T01:00:00.4409591Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4409828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4409901Z res = mod(**inputs) 2025-10-10T01:00:00.4410237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4410311Z outputs = self.mobilebert( 2025-10-10T01:00:00.4410633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4410718Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4411033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4411117Z layer_outputs = layer_module( 2025-10-10T01:00:00.4411430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4411624Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4411951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4412071Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4412395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4412506Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4412816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4412915Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4412919Z 2025-10-10T01:00:00.4413028Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4413267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4413338Z res = mod(**inputs) 2025-10-10T01:00:00.4413655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4413732Z outputs = self.mobilebert( 2025-10-10T01:00:00.4414029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4414115Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4414410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4414497Z layer_outputs = layer_module( 2025-10-10T01:00:00.4414800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4414905Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4415208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4415288Z self_outputs = self.self( 2025-10-10T01:00:00.4415599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4415684Z self.query(query_tensor) 2025-10-10T01:00:00.4415688Z 2025-10-10T01:00:00.4415809Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4416022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4416093Z res = mod(**inputs) 2025-10-10T01:00:00.4416415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4416492Z outputs = self.mobilebert( 2025-10-10T01:00:00.4416806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4416884Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4417193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4417334Z layer_outputs = layer_module( 2025-10-10T01:00:00.4417637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4417737Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4418050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4418134Z self_outputs = self.self( 2025-10-10T01:00:00.4418463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4418541Z self.key(key_tensor) 2025-10-10T01:00:00.4418545Z 2025-10-10T01:00:00.4418643Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4418731Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4418852Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4419088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4419175Z res = mod(**inputs) 2025-10-10T01:00:00.4419487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4419565Z outputs = self.mobilebert( 2025-10-10T01:00:00.4419884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4419966Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4420291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4420371Z layer_outputs = layer_module( 2025-10-10T01:00:00.4420677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4420783Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4421092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4421235Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4421547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4421640Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4421652Z 2025-10-10T01:00:00.4421767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4421981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4422061Z res = mod(**inputs) 2025-10-10T01:00:00.4422375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4422462Z outputs = self.mobilebert( 2025-10-10T01:00:00.4422769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4422847Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4423160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4423239Z layer_outputs = layer_module( 2025-10-10T01:00:00.4423562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4423656Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4423964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4424111Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4424440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4424588Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4424893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4425002Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4425005Z 2025-10-10T01:00:00.4425118Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4425349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4425430Z res = mod(**inputs) 2025-10-10T01:00:00.4425738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4425823Z outputs = self.mobilebert( 2025-10-10T01:00:00.4426162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4426242Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4426636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4426721Z layer_outputs = layer_module( 2025-10-10T01:00:00.4427036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4427146Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4427461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4427590Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4427902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4428009Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4428014Z 2025-10-10T01:00:00.4428129Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4428354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4428426Z res = mod(**inputs) 2025-10-10T01:00:00.4428733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4428821Z outputs = self.mobilebert( 2025-10-10T01:00:00.4429127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4429215Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4429525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4429615Z layer_outputs = layer_module( 2025-10-10T01:00:00.4429922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4430028Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4430345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4430469Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4430788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4430913Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4430917Z 2025-10-10T01:00:00.4431037Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4431273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4431347Z res = mod(**inputs) 2025-10-10T01:00:00.4431888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4431972Z outputs = self.mobilebert( 2025-10-10T01:00:00.4432287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4432367Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4432716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4432806Z layer_outputs = layer_module( 2025-10-10T01:00:00.4433110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4433271Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4433585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4433731Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4434028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4434119Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4434123Z 2025-10-10T01:00:00.4434244Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4434451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4434530Z res = mod(**inputs) 2025-10-10T01:00:00.4434826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4434907Z outputs = self.mobilebert( 2025-10-10T01:00:00.4435214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4435291Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4435597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4435672Z layer_outputs = layer_module( 2025-10-10T01:00:00.4435980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4436080Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4436377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4436518Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4436819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4436956Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4437251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4437351Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4437361Z 2025-10-10T01:00:00.4437472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4437682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4437757Z res = mod(**inputs) 2025-10-10T01:00:00.4438050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4438162Z outputs = self.mobilebert( 2025-10-10T01:00:00.4438465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4438541Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4438843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4438919Z layer_outputs = layer_module( 2025-10-10T01:00:00.4439237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4439339Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4439636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4439764Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4440076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4440201Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4440205Z 2025-10-10T01:00:00.4440316Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4440530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4440601Z res = mod(**inputs) 2025-10-10T01:00:00.4440901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4440993Z outputs = self.mobilebert( 2025-10-10T01:00:00.4441277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4441359Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4441642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4441716Z layer_outputs = layer_module( 2025-10-10T01:00:00.4442008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4442103Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4442392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4442507Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4442798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4442914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4442917Z 2025-10-10T01:00:00.4443023Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4443235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4443301Z res = mod(**inputs) 2025-10-10T01:00:00.4443592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4443663Z outputs = self.mobilebert( 2025-10-10T01:00:00.4443958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4444045Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4444354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4444435Z layer_outputs = layer_module( 2025-10-10T01:00:00.4444717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4444839Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4445119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4445245Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4445531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4445615Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4445635Z 2025-10-10T01:00:00.4445748Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4445945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4446012Z res = mod(**inputs) 2025-10-10T01:00:00.4446323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4446409Z outputs = self.mobilebert( 2025-10-10T01:00:00.4446688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4446759Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4447039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4447109Z layer_outputs = layer_module( 2025-10-10T01:00:00.4447384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4447491Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4447786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4447931Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4448233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4448358Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4448647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4448742Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4448746Z 2025-10-10T01:00:00.4448859Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4449058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4449134Z res = mod(**inputs) 2025-10-10T01:00:00.4449412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4449488Z outputs = self.mobilebert( 2025-10-10T01:00:00.4449775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4449849Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4450149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4450219Z layer_outputs = layer_module( 2025-10-10T01:00:00.4450495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4450594Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4450865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4450981Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4451274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4451364Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4451367Z 2025-10-10T01:00:00.4451467Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4451657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4451729Z res = mod(**inputs) 2025-10-10T01:00:00.4452014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4452092Z outputs = self.mobilebert( 2025-10-10T01:00:00.4452361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4452441Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4452734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4452820Z layer_outputs = layer_module( 2025-10-10T01:00:00.4453115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4453206Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4453483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4453594Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4453867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4453984Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4453988Z 2025-10-10T01:00:00.4454092Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4454297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4454363Z res = mod(**inputs) 2025-10-10T01:00:00.4454651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4454723Z outputs = self.mobilebert( 2025-10-10T01:00:00.4455001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4455084Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4455365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4455445Z layer_outputs = layer_module( 2025-10-10T01:00:00.4455728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4455825Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4456120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4456253Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4456562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4456662Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4456666Z 2025-10-10T01:00:00.4456775Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4456972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4457038Z res = mod(**inputs) 2025-10-10T01:00:00.4457335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4457433Z outputs = self.mobilebert( 2025-10-10T01:00:00.4457736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4457813Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4458115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4458202Z layer_outputs = layer_module( 2025-10-10T01:00:00.4458516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4458627Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4458935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4459108Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4459405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4459538Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4459839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4459940Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4459944Z 2025-10-10T01:00:00.4460063Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4460272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4460342Z res = mod(**inputs) 2025-10-10T01:00:00.4460645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4460725Z outputs = self.mobilebert( 2025-10-10T01:00:00.4461033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4461109Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4461410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4461485Z layer_outputs = layer_module( 2025-10-10T01:00:00.4461782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4461923Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4462229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4462328Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4462336Z 2025-10-10T01:00:00.4462447Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4462664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4462734Z res = mod(**inputs) 2025-10-10T01:00:00.4463038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4463122Z outputs = self.mobilebert( 2025-10-10T01:00:00.4463429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4463512Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4463805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4463882Z layer_outputs = layer_module( 2025-10-10T01:00:00.4464205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4464334Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4464636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4464757Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4464761Z 2025-10-10T01:00:00.4464874Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4465098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4465170Z res = mod(**inputs) 2025-10-10T01:00:00.4465483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4465559Z outputs = self.mobilebert( 2025-10-10T01:00:00.4465917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4465997Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4466303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4466602Z layer_outputs = layer_module( 2025-10-10T01:00:00.4466910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4467094Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4467398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4467520Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4467525Z 2025-10-10T01:00:00.4467640Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4467853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4467930Z res = mod(**inputs) 2025-10-10T01:00:00.4468226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4468310Z outputs = self.mobilebert( 2025-10-10T01:00:00.4468610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4468688Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4468994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4469070Z layer_outputs = layer_module( 2025-10-10T01:00:00.4469378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4469549Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4469856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4469986Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4470281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4470389Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4470393Z 2025-10-10T01:00:00.4470502Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4470716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4470784Z res = mod(**inputs) 2025-10-10T01:00:00.4471107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4471188Z outputs = self.mobilebert( 2025-10-10T01:00:00.4471472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4471549Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4471818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4471922Z layer_outputs = layer_module( 2025-10-10T01:00:00.4472187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4472336Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4472624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4472758Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4473031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4473117Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4473120Z 2025-10-10T01:00:00.4473229Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4473424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4473489Z res = mod(**inputs) 2025-10-10T01:00:00.4473767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4473837Z outputs = self.mobilebert( 2025-10-10T01:00:00.4474120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4474193Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4474467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4474545Z layer_outputs = layer_module( 2025-10-10T01:00:00.4474818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4474979Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4475253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4475381Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4475657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4475780Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4476059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4476152Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4476155Z 2025-10-10T01:00:00.4476263Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4476457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4476521Z res = mod(**inputs) 2025-10-10T01:00:00.4476797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4476867Z outputs = self.mobilebert( 2025-10-10T01:00:00.4477147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4477240Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4477523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4477593Z layer_outputs = layer_module( 2025-10-10T01:00:00.4477866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4478046Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4478319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4478434Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4478733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4478839Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4478851Z 2025-10-10T01:00:00.4478961Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4479174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4479260Z res = mod(**inputs) 2025-10-10T01:00:00.4479551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4479632Z outputs = self.mobilebert( 2025-10-10T01:00:00.4479922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4479994Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4480291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4480367Z layer_outputs = layer_module( 2025-10-10T01:00:00.4480663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4480752Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4481042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4481122Z self_outputs = self.self( 2025-10-10T01:00:00.4481413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4481493Z self.value(value_tensor) 2025-10-10T01:00:00.4481497Z 2025-10-10T01:00:00.4481602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4481818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4481893Z res = mod(**inputs) 2025-10-10T01:00:00.4482199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4482280Z outputs = self.mobilebert( 2025-10-10T01:00:00.4482587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4482668Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4482958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4483030Z layer_outputs = layer_module( 2025-10-10T01:00:00.4483327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4483489Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4483813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4483931Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4484238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4484327Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4484331Z 2025-10-10T01:00:00.4484440Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4484680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4484750Z res = mod(**inputs) 2025-10-10T01:00:00.4485059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4485133Z outputs = self.mobilebert( 2025-10-10T01:00:00.4485460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4485545Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4485839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4485921Z layer_outputs = layer_module( 2025-10-10T01:00:00.4486228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4486402Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4486698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4486815Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4487123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4487219Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4487527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4487626Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4487630Z 2025-10-10T01:00:00.4487746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4487958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4488026Z res = mod(**inputs) 2025-10-10T01:00:00.4488332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4488408Z outputs = self.mobilebert( 2025-10-10T01:00:00.4488719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4488801Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4489117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4489203Z layer_outputs = layer_module( 2025-10-10T01:00:00.4489523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4489625Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4489931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4490008Z self_outputs = self.self( 2025-10-10T01:00:00.4490317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4490417Z self.query(query_tensor) 2025-10-10T01:00:00.4490421Z 2025-10-10T01:00:00.4490539Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4490748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4490826Z res = mod(**inputs) 2025-10-10T01:00:00.4491133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4491208Z outputs = self.mobilebert( 2025-10-10T01:00:00.4491546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4491626Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4491940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4492019Z layer_outputs = layer_module( 2025-10-10T01:00:00.4492357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4492459Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4492766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4492850Z self_outputs = self.self( 2025-10-10T01:00:00.4493156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4493237Z self.key(key_tensor) 2025-10-10T01:00:00.4493240Z 2025-10-10T01:00:00.4493329Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4493416Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4493532Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4493761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4493841Z res = mod(**inputs) 2025-10-10T01:00:00.4494145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4494219Z outputs = self.mobilebert( 2025-10-10T01:00:00.4494529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4494606Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4494916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4494992Z layer_outputs = layer_module( 2025-10-10T01:00:00.4495298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4495401Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4495710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4495850Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4496154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4496254Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4496258Z 2025-10-10T01:00:00.4496369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4496598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4496676Z res = mod(**inputs) 2025-10-10T01:00:00.4496982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4497082Z outputs = self.mobilebert( 2025-10-10T01:00:00.4497437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4497514Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4497826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4497904Z layer_outputs = layer_module( 2025-10-10T01:00:00.4498232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4498328Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4498650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4498785Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4499112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4499276Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4499600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4499709Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4499713Z 2025-10-10T01:00:00.4499827Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4500055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4500126Z res = mod(**inputs) 2025-10-10T01:00:00.4500442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4500527Z outputs = self.mobilebert( 2025-10-10T01:00:00.4500843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4500931Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4501239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4501317Z layer_outputs = layer_module( 2025-10-10T01:00:00.4501643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4501750Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4502067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4502191Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4502517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4502611Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4502615Z 2025-10-10T01:00:00.4502728Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4502951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4503021Z res = mod(**inputs) 2025-10-10T01:00:00.4503343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4503424Z outputs = self.mobilebert( 2025-10-10T01:00:00.4503735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4503822Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4504143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4504250Z layer_outputs = layer_module( 2025-10-10T01:00:00.4504558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4504670Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4504975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4505101Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4505430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4505558Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4505562Z 2025-10-10T01:00:00.4505681Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4505914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4506002Z res = mod(**inputs) 2025-10-10T01:00:00.4506390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4506480Z outputs = self.mobilebert( 2025-10-10T01:00:00.4506794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4506875Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4507190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4507269Z layer_outputs = layer_module( 2025-10-10T01:00:00.4507577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4507697Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4508003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4508153Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4508460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4508554Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4508566Z 2025-10-10T01:00:00.4508682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4508899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4508979Z res = mod(**inputs) 2025-10-10T01:00:00.4509287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4509379Z outputs = self.mobilebert( 2025-10-10T01:00:00.4509687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4509767Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4510082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4510161Z layer_outputs = layer_module( 2025-10-10T01:00:00.4510473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4510575Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4510884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4511031Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4511364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4511509Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4511816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4511925Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4511929Z 2025-10-10T01:00:00.4512040Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4512274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4512357Z res = mod(**inputs) 2025-10-10T01:00:00.4512666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4512753Z outputs = self.mobilebert( 2025-10-10T01:00:00.4513090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4513170Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4513482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4513560Z layer_outputs = layer_module( 2025-10-10T01:00:00.4513873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4513976Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4514287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4514412Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4514722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4514826Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4514830Z 2025-10-10T01:00:00.4514942Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4515164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4515236Z res = mod(**inputs) 2025-10-10T01:00:00.4515540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4515625Z outputs = self.mobilebert( 2025-10-10T01:00:00.4515929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4516015Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4516320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4516412Z layer_outputs = layer_module( 2025-10-10T01:00:00.4516718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4516824Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4517137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4517262Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4517577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4517701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4517705Z 2025-10-10T01:00:00.4517824Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4518078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4518150Z res = mod(**inputs) 2025-10-10T01:00:00.4518463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4518542Z outputs = self.mobilebert( 2025-10-10T01:00:00.4518855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4518956Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4519262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4519350Z layer_outputs = layer_module( 2025-10-10T01:00:00.4519653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4519836Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4520135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4520273Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4520570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4520660Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4520663Z 2025-10-10T01:00:00.4520782Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4520988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4521065Z res = mod(**inputs) 2025-10-10T01:00:00.4521363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4521443Z outputs = self.mobilebert( 2025-10-10T01:00:00.4521750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4521827Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4522135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4522211Z layer_outputs = layer_module( 2025-10-10T01:00:00.4522518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4522618Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4522915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4523062Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4523370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4523512Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4523818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4523921Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4523932Z 2025-10-10T01:00:00.4524050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4524265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4524343Z res = mod(**inputs) 2025-10-10T01:00:00.4524657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4524766Z outputs = self.mobilebert( 2025-10-10T01:00:00.4525081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4525158Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4525463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4525540Z layer_outputs = layer_module( 2025-10-10T01:00:00.4525868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4525969Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4526271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4526399Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4526731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4526835Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4526839Z 2025-10-10T01:00:00.4526953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4527178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4527251Z res = mod(**inputs) 2025-10-10T01:00:00.4527566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4527652Z outputs = self.mobilebert( 2025-10-10T01:00:00.4527963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4528060Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4528364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4528441Z layer_outputs = layer_module( 2025-10-10T01:00:00.4528753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4528853Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4529166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4529287Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4529605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4529724Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4529728Z 2025-10-10T01:00:00.4529839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4530063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4530132Z res = mod(**inputs) 2025-10-10T01:00:00.4530440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4530516Z outputs = self.mobilebert( 2025-10-10T01:00:00.4530818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4530904Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4531204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4531288Z layer_outputs = layer_module( 2025-10-10T01:00:00.4531760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4531941Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4532236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4532370Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4532678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4532797Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4532801Z 2025-10-10T01:00:00.4532922Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4533132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4533204Z res = mod(**inputs) 2025-10-10T01:00:00.4533531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4533632Z outputs = self.mobilebert( 2025-10-10T01:00:00.4533934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4534012Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4534318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4534394Z layer_outputs = layer_module( 2025-10-10T01:00:00.4534694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4534803Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4535099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4535244Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4535542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4535680Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4535973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4536067Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4536070Z 2025-10-10T01:00:00.4536183Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4536385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4536456Z res = mod(**inputs) 2025-10-10T01:00:00.4536734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4536811Z outputs = self.mobilebert( 2025-10-10T01:00:00.4537101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4537179Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4537482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4537558Z layer_outputs = layer_module( 2025-10-10T01:00:00.4537862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4537992Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4538287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4538384Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4538414Z 2025-10-10T01:00:00.4538527Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4538742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4538811Z res = mod(**inputs) 2025-10-10T01:00:00.4539106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4539187Z outputs = self.mobilebert( 2025-10-10T01:00:00.4539501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4539586Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4539881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4539963Z layer_outputs = layer_module( 2025-10-10T01:00:00.4540302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4540438Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4540749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4540873Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4540877Z 2025-10-10T01:00:00.4540998Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4541216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4541287Z res = mod(**inputs) 2025-10-10T01:00:00.4541600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4541678Z outputs = self.mobilebert( 2025-10-10T01:00:00.4541998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4542079Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4542393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4542472Z layer_outputs = layer_module( 2025-10-10T01:00:00.4542778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4542962Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4543266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4543381Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4543388Z 2025-10-10T01:00:00.4543506Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4543721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4543800Z res = mod(**inputs) 2025-10-10T01:00:00.4544106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4544194Z outputs = self.mobilebert( 2025-10-10T01:00:00.4544504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4544590Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4544896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4544974Z layer_outputs = layer_module( 2025-10-10T01:00:00.4545290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4545485Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4545809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4545944Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4546257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4546535Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4546549Z 2025-10-10T01:00:00.4546675Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4546902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4546975Z res = mod(**inputs) 2025-10-10T01:00:00.4547315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4547411Z outputs = self.mobilebert( 2025-10-10T01:00:00.4547713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4547799Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4548083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4548166Z layer_outputs = layer_module( 2025-10-10T01:00:00.4548447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4548604Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4548892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4549017Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4549305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4549393Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4549396Z 2025-10-10T01:00:00.4549507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4549706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4549773Z res = mod(**inputs) 2025-10-10T01:00:00.4550060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4550132Z outputs = self.mobilebert( 2025-10-10T01:00:00.4550421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4550496Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4550783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4550856Z layer_outputs = layer_module( 2025-10-10T01:00:00.4551136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4551299Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4551580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4551709Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4551991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4552135Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4552431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4552522Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4552526Z 2025-10-10T01:00:00.4552634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4552846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4552918Z res = mod(**inputs) 2025-10-10T01:00:00.4553193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4553262Z outputs = self.mobilebert( 2025-10-10T01:00:00.4553561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4553649Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4553929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4553998Z layer_outputs = layer_module( 2025-10-10T01:00:00.4554271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4554436Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4554712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4554830Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4555105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4555196Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4555200Z 2025-10-10T01:00:00.4555298Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4555501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4555576Z res = mod(**inputs) 2025-10-10T01:00:00.4555878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4555961Z outputs = self.mobilebert( 2025-10-10T01:00:00.4556267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4556344Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4556659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4556739Z layer_outputs = layer_module( 2025-10-10T01:00:00.4557052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4557146Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4557462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4557539Z self_outputs = self.self( 2025-10-10T01:00:00.4557847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4557934Z self.value(value_tensor) 2025-10-10T01:00:00.4557938Z 2025-10-10T01:00:00.4558046Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4558264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4558354Z res = mod(**inputs) 2025-10-10T01:00:00.4558628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4558704Z outputs = self.mobilebert( 2025-10-10T01:00:00.4558977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4559055Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4559343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4559424Z layer_outputs = layer_module( 2025-10-10T01:00:00.4559701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4559859Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4560175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4560286Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4560574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4560657Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4560661Z 2025-10-10T01:00:00.4560770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4560971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4561038Z res = mod(**inputs) 2025-10-10T01:00:00.4561327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4561399Z outputs = self.mobilebert( 2025-10-10T01:00:00.4561699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4561770Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4562042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4562119Z layer_outputs = layer_module( 2025-10-10T01:00:00.4562396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4562558Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4562835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4562952Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4563235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4563324Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4563611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4563705Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4563709Z 2025-10-10T01:00:00.4563819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4564018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4564083Z res = mod(**inputs) 2025-10-10T01:00:00.4564369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4564439Z outputs = self.mobilebert( 2025-10-10T01:00:00.4564726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4564816Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4565110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4565182Z layer_outputs = layer_module( 2025-10-10T01:00:00.4565470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4565582Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4565863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4565943Z self_outputs = self.self( 2025-10-10T01:00:00.4566221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4566336Z self.query(query_tensor) 2025-10-10T01:00:00.4566340Z 2025-10-10T01:00:00.4566454Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4566651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4566726Z res = mod(**inputs) 2025-10-10T01:00:00.4567033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4567107Z outputs = self.mobilebert( 2025-10-10T01:00:00.4567415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4567492Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4567799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4567876Z layer_outputs = layer_module( 2025-10-10T01:00:00.4568183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4568275Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4568573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4568657Z self_outputs = self.self( 2025-10-10T01:00:00.4568954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4569035Z self.key(key_tensor) 2025-10-10T01:00:00.4569038Z 2025-10-10T01:00:00.4569127Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4569213Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4569338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4569535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4569611Z res = mod(**inputs) 2025-10-10T01:00:00.4569897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4569977Z outputs = self.mobilebert( 2025-10-10T01:00:00.4570275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4570352Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4570656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4570732Z layer_outputs = layer_module( 2025-10-10T01:00:00.4571035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4571146Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4571444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4571583Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4571882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4571978Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4571983Z 2025-10-10T01:00:00.4572106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4572321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4572392Z res = mod(**inputs) 2025-10-10T01:00:00.4572693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4572782Z outputs = self.mobilebert( 2025-10-10T01:00:00.4573109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4573195Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4573490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4573566Z layer_outputs = layer_module( 2025-10-10T01:00:00.4573875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4573966Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4574272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4574403Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4574717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4574855Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4575153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4575262Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4575266Z 2025-10-10T01:00:00.4575376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4575596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4575667Z res = mod(**inputs) 2025-10-10T01:00:00.4575961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4576045Z outputs = self.mobilebert( 2025-10-10T01:00:00.4576344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4576430Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4576727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4576812Z layer_outputs = layer_module( 2025-10-10T01:00:00.4577107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4577212Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4577515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4577636Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4577942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4578056Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4578060Z 2025-10-10T01:00:00.4578168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4578382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4578450Z res = mod(**inputs) 2025-10-10T01:00:00.4578756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4578845Z outputs = self.mobilebert( 2025-10-10T01:00:00.4579153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4579230Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4579545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4579647Z layer_outputs = layer_module( 2025-10-10T01:00:00.4579941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4580051Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4580347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4580466Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4580773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4580893Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4580897Z 2025-10-10T01:00:00.4581015Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4581227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4581304Z res = mod(**inputs) 2025-10-10T01:00:00.4581599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4581675Z outputs = self.mobilebert( 2025-10-10T01:00:00.4581980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4582056Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4582370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4582447Z layer_outputs = layer_module( 2025-10-10T01:00:00.4582743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4582855Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4583154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4583295Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4583594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4583690Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4583693Z 2025-10-10T01:00:00.4583804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4584013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4584091Z res = mod(**inputs) 2025-10-10T01:00:00.4584397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4584505Z outputs = self.mobilebert( 2025-10-10T01:00:00.4584814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4584891Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4585214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4585290Z layer_outputs = layer_module( 2025-10-10T01:00:00.4585621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4585726Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4586047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4586183Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4586608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4586762Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4587080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4587192Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4587196Z 2025-10-10T01:00:00.4587308Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4587534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4587607Z res = mod(**inputs) 2025-10-10T01:00:00.4587916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4588005Z outputs = self.mobilebert( 2025-10-10T01:00:00.4588318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4588406Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4588714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4588792Z layer_outputs = layer_module( 2025-10-10T01:00:00.4589108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4589212Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4589526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4589650Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4589971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4590067Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4590071Z 2025-10-10T01:00:00.4590183Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4590409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4590481Z res = mod(**inputs) 2025-10-10T01:00:00.4590794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4590872Z outputs = self.mobilebert( 2025-10-10T01:00:00.4591181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4591270Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4591580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4591689Z layer_outputs = layer_module( 2025-10-10T01:00:00.4591996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4592107Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4592413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4592564Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4592878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4593003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4593007Z 2025-10-10T01:00:00.4593128Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4593381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4593447Z res = mod(**inputs) 2025-10-10T01:00:00.4593731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4593801Z outputs = self.mobilebert( 2025-10-10T01:00:00.4594082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4594155Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4594439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4594521Z layer_outputs = layer_module( 2025-10-10T01:00:00.4594787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4594888Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4595156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4595284Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4595555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4595637Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4595649Z 2025-10-10T01:00:00.4595753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4595946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4596017Z res = mod(**inputs) 2025-10-10T01:00:00.4596295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4596378Z outputs = self.mobilebert( 2025-10-10T01:00:00.4596654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4596725Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4597014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4597085Z layer_outputs = layer_module( 2025-10-10T01:00:00.4597369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4597462Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4597750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4597883Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4598183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4598314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4598595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4598698Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4598701Z 2025-10-10T01:00:00.4598817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4599016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4599091Z res = mod(**inputs) 2025-10-10T01:00:00.4599370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4599498Z outputs = self.mobilebert( 2025-10-10T01:00:00.4599770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4599841Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4600120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4600191Z layer_outputs = layer_module( 2025-10-10T01:00:00.4600470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4600563Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4600845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4600959Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4601244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4601338Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4601341Z 2025-10-10T01:00:00.4601446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4601652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4601717Z res = mod(**inputs) 2025-10-10T01:00:00.4602000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4602081Z outputs = self.mobilebert( 2025-10-10T01:00:00.4602377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4602462Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4602771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4602852Z layer_outputs = layer_module( 2025-10-10T01:00:00.4603130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4603226Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4603510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4603623Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4603913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4604032Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4604036Z 2025-10-10T01:00:00.4604151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4604381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4604450Z res = mod(**inputs) 2025-10-10T01:00:00.4604753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4604829Z outputs = self.mobilebert( 2025-10-10T01:00:00.4605131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4605222Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4605521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4605602Z layer_outputs = layer_module( 2025-10-10T01:00:00.4605915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4606039Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4606338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4606475Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4606771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4606860Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4606865Z 2025-10-10T01:00:00.4606982Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4607197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4607269Z res = mod(**inputs) 2025-10-10T01:00:00.4607549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4607622Z outputs = self.mobilebert( 2025-10-10T01:00:00.4607922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4607998Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4608300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4608375Z layer_outputs = layer_module( 2025-10-10T01:00:00.4608679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4608778Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4609075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4609220Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4609516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4609654Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4609950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4610050Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4610061Z 2025-10-10T01:00:00.4610173Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4610383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4610458Z res = mod(**inputs) 2025-10-10T01:00:00.4610737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4610840Z outputs = self.mobilebert( 2025-10-10T01:00:00.4611135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4611213Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4611512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4611590Z layer_outputs = layer_module( 2025-10-10T01:00:00.4611908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4612041Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4612342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4612441Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4612479Z 2025-10-10T01:00:00.4612592Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4612808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4612879Z res = mod(**inputs) 2025-10-10T01:00:00.4613183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4613259Z outputs = self.mobilebert( 2025-10-10T01:00:00.4613555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4613639Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4613933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4614016Z layer_outputs = layer_module( 2025-10-10T01:00:00.4614317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4614446Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4614749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4614867Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4614871Z 2025-10-10T01:00:00.4614987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4615197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4615272Z res = mod(**inputs) 2025-10-10T01:00:00.4615569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4615645Z outputs = self.mobilebert( 2025-10-10T01:00:00.4615953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4616030Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4616335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4616412Z layer_outputs = layer_module( 2025-10-10T01:00:00.4616714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4616900Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4617204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4617318Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4617322Z 2025-10-10T01:00:00.4617465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4617690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4617762Z res = mod(**inputs) 2025-10-10T01:00:00.4618071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4618156Z outputs = self.mobilebert( 2025-10-10T01:00:00.4618464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4618575Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4618885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4618964Z layer_outputs = layer_module( 2025-10-10T01:00:00.4620252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4620484Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4620800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4620935Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4621250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4621354Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4621358Z 2025-10-10T01:00:00.4621474Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4621697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4621770Z res = mod(**inputs) 2025-10-10T01:00:00.4622091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4622172Z outputs = self.mobilebert( 2025-10-10T01:00:00.4622488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4622568Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4622877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4622965Z layer_outputs = layer_module( 2025-10-10T01:00:00.4623273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4623451Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4623759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4623898Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4624213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4624306Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4624310Z 2025-10-10T01:00:00.4624431Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4624650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4624728Z res = mod(**inputs) 2025-10-10T01:00:00.4625035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4625114Z outputs = self.mobilebert( 2025-10-10T01:00:00.4625428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4625527Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4625840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4625918Z layer_outputs = layer_module( 2025-10-10T01:00:00.4626223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4626501Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4626820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4626965Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4627327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4627494Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4627805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4627908Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4627912Z 2025-10-10T01:00:00.4628038Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4628257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4628336Z res = mod(**inputs) 2025-10-10T01:00:00.4628644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4628723Z outputs = self.mobilebert( 2025-10-10T01:00:00.4629042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4629125Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4629444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4629522Z layer_outputs = layer_module( 2025-10-10T01:00:00.4629836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4630015Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4630327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4630456Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4630770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4630873Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4630877Z 2025-10-10T01:00:00.4630989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4631212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4631285Z res = mod(**inputs) 2025-10-10T01:00:00.4631747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4631843Z outputs = self.mobilebert( 2025-10-10T01:00:00.4632154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4632243Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4632554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4632682Z layer_outputs = layer_module( 2025-10-10T01:00:00.4632998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4633096Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4633416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4633496Z self_outputs = self.self( 2025-10-10T01:00:00.4633828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4633912Z self.value(value_tensor) 2025-10-10T01:00:00.4633917Z 2025-10-10T01:00:00.4634030Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4634251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4634350Z res = mod(**inputs) 2025-10-10T01:00:00.4634690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4634771Z outputs = self.mobilebert( 2025-10-10T01:00:00.4635078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4635165Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4635471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4635557Z layer_outputs = layer_module( 2025-10-10T01:00:00.4635865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4636048Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4636370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4636480Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4636762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4636844Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4636848Z 2025-10-10T01:00:00.4636955Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4637149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4637215Z res = mod(**inputs) 2025-10-10T01:00:00.4637496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4637565Z outputs = self.mobilebert( 2025-10-10T01:00:00.4637849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4637919Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4638197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4638267Z layer_outputs = layer_module( 2025-10-10T01:00:00.4638538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4638701Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4638976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4639091Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4639377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4639464Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4639744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4639834Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4639837Z 2025-10-10T01:00:00.4639946Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4640151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4640224Z res = mod(**inputs) 2025-10-10T01:00:00.4640499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4640569Z outputs = self.mobilebert( 2025-10-10T01:00:00.4640862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4640960Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4641240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4641310Z layer_outputs = layer_module( 2025-10-10T01:00:00.4641585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4641679Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4641952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4642031Z self_outputs = self.self( 2025-10-10T01:00:00.4642306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4642386Z self.query(query_tensor) 2025-10-10T01:00:00.4642390Z 2025-10-10T01:00:00.4642492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4642688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4642760Z res = mod(**inputs) 2025-10-10T01:00:00.4643034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4643110Z outputs = self.mobilebert( 2025-10-10T01:00:00.4643386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4643457Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4643737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4643809Z layer_outputs = layer_module( 2025-10-10T01:00:00.4644091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4644176Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4644459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4644529Z self_outputs = self.self( 2025-10-10T01:00:00.4644811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4644886Z self.key(key_tensor) 2025-10-10T01:00:00.4644890Z 2025-10-10T01:00:00.4644972Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4645061Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4645163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4645360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4645451Z res = mod(**inputs) 2025-10-10T01:00:00.4645731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4645809Z outputs = self.mobilebert( 2025-10-10T01:00:00.4646085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4646158Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4646459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4646531Z layer_outputs = layer_module( 2025-10-10T01:00:00.4646812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4646896Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4647210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4647335Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4647611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4647704Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4647708Z 2025-10-10T01:00:00.4647811Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4648011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4648076Z res = mod(**inputs) 2025-10-10T01:00:00.4648353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4648431Z outputs = self.mobilebert( 2025-10-10T01:00:00.4648706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4648784Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4649055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4649132Z layer_outputs = layer_module( 2025-10-10T01:00:00.4649407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4649503Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4649773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4649887Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4650155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4650279Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4650546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4650643Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4650646Z 2025-10-10T01:00:00.4650746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4650942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4651005Z res = mod(**inputs) 2025-10-10T01:00:00.4651280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4651348Z outputs = self.mobilebert( 2025-10-10T01:00:00.4651627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4651708Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4651973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4652052Z layer_outputs = layer_module( 2025-10-10T01:00:00.4652322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4652444Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4652728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4652839Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4653125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4653226Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4653230Z 2025-10-10T01:00:00.4653340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4653537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4653600Z res = mod(**inputs) 2025-10-10T01:00:00.4653885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4653952Z outputs = self.mobilebert( 2025-10-10T01:00:00.4654215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4654282Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4654542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4654617Z layer_outputs = layer_module( 2025-10-10T01:00:00.4654877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4654973Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4655235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4655351Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4655617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4655725Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4655730Z 2025-10-10T01:00:00.4655836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4656025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4656096Z res = mod(**inputs) 2025-10-10T01:00:00.4656362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4656430Z outputs = self.mobilebert( 2025-10-10T01:00:00.4656699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4656770Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4657049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4657118Z layer_outputs = layer_module( 2025-10-10T01:00:00.4657395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4657507Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4657783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4657916Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4658187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4658277Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4658280Z 2025-10-10T01:00:00.4658393Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4658593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4658658Z res = mod(**inputs) 2025-10-10T01:00:00.4658930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4659038Z outputs = self.mobilebert( 2025-10-10T01:00:00.4659309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4659387Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4659658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4659727Z layer_outputs = layer_module( 2025-10-10T01:00:00.4660006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4660098Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4660376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4660503Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4660794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4660917Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4661195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4661297Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4661301Z 2025-10-10T01:00:00.4661405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4661610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4661676Z res = mod(**inputs) 2025-10-10T01:00:00.4661952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4662032Z outputs = self.mobilebert( 2025-10-10T01:00:00.4662313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4662393Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4662670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4662748Z layer_outputs = layer_module( 2025-10-10T01:00:00.4663026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4663124Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4663410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4663522Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4663809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4663920Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4663924Z 2025-10-10T01:00:00.4664032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4664252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4664322Z res = mod(**inputs) 2025-10-10T01:00:00.4664650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4664728Z outputs = self.mobilebert( 2025-10-10T01:00:00.4665032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4665109Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4665444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4665546Z layer_outputs = layer_module( 2025-10-10T01:00:00.4665839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4665947Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4666242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4666431Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4666755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4666878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4666882Z 2025-10-10T01:00:00.4667005Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4667227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4667308Z res = mod(**inputs) 2025-10-10T01:00:00.4667615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4667695Z outputs = self.mobilebert( 2025-10-10T01:00:00.4668012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4668091Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4668406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4668482Z layer_outputs = layer_module( 2025-10-10T01:00:00.4668776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4668891Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4669188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4669331Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4669642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4669739Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4669743Z 2025-10-10T01:00:00.4669854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4670062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4670138Z res = mod(**inputs) 2025-10-10T01:00:00.4670435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4670541Z outputs = self.mobilebert( 2025-10-10T01:00:00.4670838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4670916Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4671229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4671305Z layer_outputs = layer_module( 2025-10-10T01:00:00.4671629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4671730Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4672042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4672174Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4672506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4672644Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4672942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4673047Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4673051Z 2025-10-10T01:00:00.4673161Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4673379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4673449Z res = mod(**inputs) 2025-10-10T01:00:00.4673747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4673833Z outputs = self.mobilebert( 2025-10-10T01:00:00.4674134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4674217Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4674517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4674593Z layer_outputs = layer_module( 2025-10-10T01:00:00.4674892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4674983Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4675267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4675379Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4675673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4675759Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4675763Z 2025-10-10T01:00:00.4675865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4676070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4676135Z res = mod(**inputs) 2025-10-10T01:00:00.4676427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4676498Z outputs = self.mobilebert( 2025-10-10T01:00:00.4676780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4676861Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4679555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4679675Z layer_outputs = layer_module( 2025-10-10T01:00:00.4679973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4680072Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4680373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4680495Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4680805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4680927Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4680931Z 2025-10-10T01:00:00.4681050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4681351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4681424Z res = mod(**inputs) 2025-10-10T01:00:00.4681733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4681812Z outputs = self.mobilebert( 2025-10-10T01:00:00.4682118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4682196Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4682490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4682570Z layer_outputs = layer_module( 2025-10-10T01:00:00.4682861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4682964Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4683238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4683362Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4683642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4683725Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4683729Z 2025-10-10T01:00:00.4683839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4684031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4684106Z res = mod(**inputs) 2025-10-10T01:00:00.4684392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4684470Z outputs = self.mobilebert( 2025-10-10T01:00:00.4684759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4684831Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4685120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4685192Z layer_outputs = layer_module( 2025-10-10T01:00:00.4685489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4685594Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4685892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4686054Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4686407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4686548Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4686845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4686948Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4686951Z 2025-10-10T01:00:00.4687071Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4687287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4687367Z res = mod(**inputs) 2025-10-10T01:00:00.4687674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4687788Z outputs = self.mobilebert( 2025-10-10T01:00:00.4688090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4688168Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4688469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4688544Z layer_outputs = layer_module( 2025-10-10T01:00:00.4688856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4688987Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4689284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4689386Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4689393Z 2025-10-10T01:00:00.4689505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4689722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4689791Z res = mod(**inputs) 2025-10-10T01:00:00.4690087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4690172Z outputs = self.mobilebert( 2025-10-10T01:00:00.4690479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4690562Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4690856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4690936Z layer_outputs = layer_module( 2025-10-10T01:00:00.4691234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4691364Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4691667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4691786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4691790Z 2025-10-10T01:00:00.4691904Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4692112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4692181Z res = mod(**inputs) 2025-10-10T01:00:00.4692494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4692569Z outputs = self.mobilebert( 2025-10-10T01:00:00.4692946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4693025Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4693328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4693404Z layer_outputs = layer_module( 2025-10-10T01:00:00.4693698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4693874Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4694167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4694275Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4694280Z 2025-10-10T01:00:00.4694407Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4694639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4694708Z res = mod(**inputs) 2025-10-10T01:00:00.4695003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4695088Z outputs = self.mobilebert( 2025-10-10T01:00:00.4695384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4695470Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4695763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4695839Z layer_outputs = layer_module( 2025-10-10T01:00:00.4696144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4696317Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4696629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4696763Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4697075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4697180Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4697184Z 2025-10-10T01:00:00.4697306Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4697541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4697610Z res = mod(**inputs) 2025-10-10T01:00:00.4697923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4697999Z outputs = self.mobilebert( 2025-10-10T01:00:00.4698295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4698379Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4698680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4698765Z layer_outputs = layer_module( 2025-10-10T01:00:00.4699060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4699232Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4699538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4699711Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4700015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4700105Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4700109Z 2025-10-10T01:00:00.4700226Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4700452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4700521Z res = mod(**inputs) 2025-10-10T01:00:00.4700831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4700906Z outputs = self.mobilebert( 2025-10-10T01:00:00.4701227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4701327Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4701637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4701712Z layer_outputs = layer_module( 2025-10-10T01:00:00.4702005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4702178Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4702483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4702622Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4702938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4703082Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4703387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4703490Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4703494Z 2025-10-10T01:00:00.4703614Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4703847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4703924Z res = mod(**inputs) 2025-10-10T01:00:00.4704228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4704305Z outputs = self.mobilebert( 2025-10-10T01:00:00.4704619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4704704Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4705013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4705090Z layer_outputs = layer_module( 2025-10-10T01:00:00.4705410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4705588Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4705902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4706032Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4706431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4706570Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4706593Z 2025-10-10T01:00:00.4706710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4706926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4707007Z res = mod(**inputs) 2025-10-10T01:00:00.4707322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4707407Z outputs = self.mobilebert( 2025-10-10T01:00:00.4707726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4707813Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4708112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4708231Z layer_outputs = layer_module( 2025-10-10T01:00:00.4708551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4708649Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4708960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4709040Z self_outputs = self.self( 2025-10-10T01:00:00.4709346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4709431Z self.value(value_tensor) 2025-10-10T01:00:00.4709436Z 2025-10-10T01:00:00.4709549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4709769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4709845Z res = mod(**inputs) 2025-10-10T01:00:00.4710185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4710262Z outputs = self.mobilebert( 2025-10-10T01:00:00.4710565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4710650Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4710954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4711037Z layer_outputs = layer_module( 2025-10-10T01:00:00.4711338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4711514Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4711838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4711960Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4712273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4712365Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4712369Z 2025-10-10T01:00:00.4712489Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4712705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4712777Z res = mod(**inputs) 2025-10-10T01:00:00.4713097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4713175Z outputs = self.mobilebert( 2025-10-10T01:00:00.4713528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4713612Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4713921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4714009Z layer_outputs = layer_module( 2025-10-10T01:00:00.4714320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4714502Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4714811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4714940Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4715265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4715382Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4715696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4715797Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4715801Z 2025-10-10T01:00:00.4715920Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4716137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4716209Z res = mod(**inputs) 2025-10-10T01:00:00.4716520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4716597Z outputs = self.mobilebert( 2025-10-10T01:00:00.4716914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4716999Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4717310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4717390Z layer_outputs = layer_module( 2025-10-10T01:00:00.4717695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4717799Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4718105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4718190Z self_outputs = self.self( 2025-10-10T01:00:00.4718495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4718578Z self.query(query_tensor) 2025-10-10T01:00:00.4718589Z 2025-10-10T01:00:00.4718705Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4718921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4719001Z res = mod(**inputs) 2025-10-10T01:00:00.4719304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4719388Z outputs = self.mobilebert( 2025-10-10T01:00:00.4719691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4719768Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4720078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4720186Z layer_outputs = layer_module( 2025-10-10T01:00:00.4720517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4720616Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4720921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4721005Z self_outputs = self.self( 2025-10-10T01:00:00.4721311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4721395Z self.key(key_tensor) 2025-10-10T01:00:00.4721399Z 2025-10-10T01:00:00.4721491Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4721586Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4721702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4721935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4722033Z res = mod(**inputs) 2025-10-10T01:00:00.4722341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4722426Z outputs = self.mobilebert( 2025-10-10T01:00:00.4722730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4722812Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4723131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4723208Z layer_outputs = layer_module( 2025-10-10T01:00:00.4723513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4723603Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4723905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4724047Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4724344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4724442Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4724446Z 2025-10-10T01:00:00.4724555Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4724768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4724837Z res = mod(**inputs) 2025-10-10T01:00:00.4725142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4725227Z outputs = self.mobilebert( 2025-10-10T01:00:00.4725537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4725624Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4725927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4726005Z layer_outputs = layer_module( 2025-10-10T01:00:00.4726319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4726412Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4726725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4726859Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4727195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4727354Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4727660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4727774Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4727777Z 2025-10-10T01:00:00.4727891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4728126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4728195Z res = mod(**inputs) 2025-10-10T01:00:00.4728501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4728576Z outputs = self.mobilebert( 2025-10-10T01:00:00.4728895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4729000Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4729310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4729394Z layer_outputs = layer_module( 2025-10-10T01:00:00.4729701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4729803Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4730118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4730237Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4730556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4730650Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4730654Z 2025-10-10T01:00:00.4730770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4730983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4731052Z res = mod(**inputs) 2025-10-10T01:00:00.4731367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4731660Z outputs = self.mobilebert( 2025-10-10T01:00:00.4731980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4732060Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4732358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4732451Z layer_outputs = layer_module( 2025-10-10T01:00:00.4732750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4732860Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4733156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4733283Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4733591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4733713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4733719Z 2025-10-10T01:00:00.4733839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4734053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4734254Z res = mod(**inputs) 2025-10-10T01:00:00.4734558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4734635Z outputs = self.mobilebert( 2025-10-10T01:00:00.4734954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4735044Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4735355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4735433Z layer_outputs = layer_module( 2025-10-10T01:00:00.4735742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4735848Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4736201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4736350Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4736647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4736747Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4736752Z 2025-10-10T01:00:00.4736863Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4737072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4737150Z res = mod(**inputs) 2025-10-10T01:00:00.4737514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4737604Z outputs = self.mobilebert( 2025-10-10T01:00:00.4737902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4737986Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4738286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4738363Z layer_outputs = layer_module( 2025-10-10T01:00:00.4738670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4738771Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4739075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4739211Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4739516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4739660Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4739959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4740067Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4740071Z 2025-10-10T01:00:00.4740180Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4740397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4740468Z res = mod(**inputs) 2025-10-10T01:00:00.4740768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4740847Z outputs = self.mobilebert( 2025-10-10T01:00:00.4741159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4741241Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4741523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4741594Z layer_outputs = layer_module( 2025-10-10T01:00:00.4741877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4741970Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4742252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4742365Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4742666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4742779Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4742783Z 2025-10-10T01:00:00.4742884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4743086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4743155Z res = mod(**inputs) 2025-10-10T01:00:00.4743460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4743534Z outputs = self.mobilebert( 2025-10-10T01:00:00.4743831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4743912Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4744211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4744299Z layer_outputs = layer_module( 2025-10-10T01:00:00.4744597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4744702Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4745001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4745120Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4745426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4745544Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4745548Z 2025-10-10T01:00:00.4745663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4745877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4745956Z res = mod(**inputs) 2025-10-10T01:00:00.4746255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4746379Z outputs = self.mobilebert( 2025-10-10T01:00:00.4746704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4746785Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4747100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4747178Z layer_outputs = layer_module( 2025-10-10T01:00:00.4747485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4747627Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4747953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4748099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4748415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4748507Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4748511Z 2025-10-10T01:00:00.4748615Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4748808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4748884Z res = mod(**inputs) 2025-10-10T01:00:00.4749158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4749275Z outputs = self.mobilebert( 2025-10-10T01:00:00.4749551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4749623Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4749903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4749974Z layer_outputs = layer_module( 2025-10-10T01:00:00.4750254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4750347Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4750627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4750749Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4751027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4751155Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4751429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4751525Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4751529Z 2025-10-10T01:00:00.4751630Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4751822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4751894Z res = mod(**inputs) 2025-10-10T01:00:00.4752165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4752244Z outputs = self.mobilebert( 2025-10-10T01:00:00.4752522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4752602Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4752873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4752943Z layer_outputs = layer_module( 2025-10-10T01:00:00.4753222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4753313Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4753592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4753702Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4754012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4754108Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4754111Z 2025-10-10T01:00:00.4754214Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4754417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4754481Z res = mod(**inputs) 2025-10-10T01:00:00.4754764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4754835Z outputs = self.mobilebert( 2025-10-10T01:00:00.4755110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4755189Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4755481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4755580Z layer_outputs = layer_module( 2025-10-10T01:00:00.4755850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4755944Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4756225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4756335Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4756617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4756728Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4756732Z 2025-10-10T01:00:00.4756839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4757039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4757103Z res = mod(**inputs) 2025-10-10T01:00:00.4757382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4757451Z outputs = self.mobilebert( 2025-10-10T01:00:00.4757729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4757800Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4758075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4758152Z layer_outputs = layer_module( 2025-10-10T01:00:00.4758426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4758539Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4758805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4758931Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4759198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4759279Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4759283Z 2025-10-10T01:00:00.4759390Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4759584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4759654Z res = mod(**inputs) 2025-10-10T01:00:00.4759928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4760021Z outputs = self.mobilebert( 2025-10-10T01:00:00.4760314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4760387Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4760664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4760734Z layer_outputs = layer_module( 2025-10-10T01:00:00.4761012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4761105Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4761377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4761524Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4761812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4761939Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4762209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4762315Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4762318Z 2025-10-10T01:00:00.4762416Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4762602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4762674Z res = mod(**inputs) 2025-10-10T01:00:00.4762940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4763018Z outputs = self.mobilebert( 2025-10-10T01:00:00.4763285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4763354Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4763626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4763696Z layer_outputs = layer_module( 2025-10-10T01:00:00.4763975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4764097Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4764376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4764460Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4764466Z 2025-10-10T01:00:00.4764570Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4764771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4764834Z res = mod(**inputs) 2025-10-10T01:00:00.4765112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4765182Z outputs = self.mobilebert( 2025-10-10T01:00:00.4765453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4765537Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4765834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4765917Z layer_outputs = layer_module( 2025-10-10T01:00:00.4766250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4766382Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4766663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4766775Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4766778Z 2025-10-10T01:00:00.4766890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4767087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4767160Z res = mod(**inputs) 2025-10-10T01:00:00.4767443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4767525Z outputs = self.mobilebert( 2025-10-10T01:00:00.4767833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4767920Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4768197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4768267Z layer_outputs = layer_module( 2025-10-10T01:00:00.4768547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4768701Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4769001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4769111Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4769115Z 2025-10-10T01:00:00.4769227Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4769459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4769522Z res = mod(**inputs) 2025-10-10T01:00:00.4769794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4769870Z outputs = self.mobilebert( 2025-10-10T01:00:00.4770144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4770224Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4770502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4770580Z layer_outputs = layer_module( 2025-10-10T01:00:00.4770868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4771025Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4771303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4771425Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4771703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4771797Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4771800Z 2025-10-10T01:00:00.4771906Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4772102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4772168Z res = mod(**inputs) 2025-10-10T01:00:00.4772472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4772560Z outputs = self.mobilebert( 2025-10-10T01:00:00.4772847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4772921Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4773202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4773283Z layer_outputs = layer_module( 2025-10-10T01:00:00.4773567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4773731Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4774035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4774183Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4774463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4774549Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4774553Z 2025-10-10T01:00:00.4774666Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4774864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4774935Z res = mod(**inputs) 2025-10-10T01:00:00.4775218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4775289Z outputs = self.mobilebert( 2025-10-10T01:00:00.4775580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4775656Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4775948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4776020Z layer_outputs = layer_module( 2025-10-10T01:00:00.4776311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4776466Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4776751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4776882Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4777168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4777303Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4777592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4777687Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4777696Z 2025-10-10T01:00:00.4777800Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4777999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4778071Z res = mod(**inputs) 2025-10-10T01:00:00.4778353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4778430Z outputs = self.mobilebert( 2025-10-10T01:00:00.4778714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4778823Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4779113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4779185Z layer_outputs = layer_module( 2025-10-10T01:00:00.4779475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4779636Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4779920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4780040Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4780325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4780453Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4780458Z 2025-10-10T01:00:00.4780565Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4780769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4780835Z res = mod(**inputs) 2025-10-10T01:00:00.4781120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4781205Z outputs = self.mobilebert( 2025-10-10T01:00:00.4781513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4781598Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4781902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4781984Z layer_outputs = layer_module( 2025-10-10T01:00:00.4782292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4782386Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4782695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4782773Z self_outputs = self.self( 2025-10-10T01:00:00.4783094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4783174Z self.value(value_tensor) 2025-10-10T01:00:00.4783178Z 2025-10-10T01:00:00.4783293Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4783533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4783606Z res = mod(**inputs) 2025-10-10T01:00:00.4783933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4784010Z outputs = self.mobilebert( 2025-10-10T01:00:00.4784323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4784411Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4784727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4784812Z layer_outputs = layer_module( 2025-10-10T01:00:00.4785120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4785300Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4785654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4785782Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4786100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4786190Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4786194Z 2025-10-10T01:00:00.4786378Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4786625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4786695Z res = mod(**inputs) 2025-10-10T01:00:00.4787017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4787094Z outputs = self.mobilebert( 2025-10-10T01:00:00.4787434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4787533Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4787857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4787935Z layer_outputs = layer_module( 2025-10-10T01:00:00.4788243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4788432Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4788718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4788839Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4789126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4789228Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4789510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4789606Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4789610Z 2025-10-10T01:00:00.4789722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4789921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4789993Z res = mod(**inputs) 2025-10-10T01:00:00.4790273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4790344Z outputs = self.mobilebert( 2025-10-10T01:00:00.4790635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4790714Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4791002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4791073Z layer_outputs = layer_module( 2025-10-10T01:00:00.4791418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4791508Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4791790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4791871Z self_outputs = self.self( 2025-10-10T01:00:00.4792152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4792252Z self.query(query_tensor) 2025-10-10T01:00:00.4792256Z 2025-10-10T01:00:00.4792411Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4792617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4792690Z res = mod(**inputs) 2025-10-10T01:00:00.4792970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4793047Z outputs = self.mobilebert( 2025-10-10T01:00:00.4793330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4793412Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4793691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4793761Z layer_outputs = layer_module( 2025-10-10T01:00:00.4794075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4794162Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4794440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4794509Z self_outputs = self.self( 2025-10-10T01:00:00.4794780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4794856Z self.key(key_tensor) 2025-10-10T01:00:00.4794860Z 2025-10-10T01:00:00.4794943Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4795030Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4795135Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4795331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4795407Z res = mod(**inputs) 2025-10-10T01:00:00.4795681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4795759Z outputs = self.mobilebert( 2025-10-10T01:00:00.4796033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4796111Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4796386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4796458Z layer_outputs = layer_module( 2025-10-10T01:00:00.4796747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4796836Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4797130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4797255Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4797538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4797633Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4797636Z 2025-10-10T01:00:00.4797746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4797965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4798036Z res = mod(**inputs) 2025-10-10T01:00:00.4798351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4798426Z outputs = self.mobilebert( 2025-10-10T01:00:00.4798753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4798839Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4799132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4799213Z layer_outputs = layer_module( 2025-10-10T01:00:00.4799491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4799577Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4799862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4799986Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4800288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4800432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4800731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4800831Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4800835Z 2025-10-10T01:00:00.4800941Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4801157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4801226Z res = mod(**inputs) 2025-10-10T01:00:00.4801527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4801603Z outputs = self.mobilebert( 2025-10-10T01:00:00.4801900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4801987Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4802282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4802366Z layer_outputs = layer_module( 2025-10-10T01:00:00.4802659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4802769Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4803065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4803186Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4803492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4803585Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4803589Z 2025-10-10T01:00:00.4803705Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4803914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4803984Z res = mod(**inputs) 2025-10-10T01:00:00.4804288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4804364Z outputs = self.mobilebert( 2025-10-10T01:00:00.4804668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4804745Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4805050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4805147Z layer_outputs = layer_module( 2025-10-10T01:00:00.4805459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4805572Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4805872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4805993Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4806276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4806391Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4806403Z 2025-10-10T01:00:00.4806507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4806721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4806815Z res = mod(**inputs) 2025-10-10T01:00:00.4807114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4807197Z outputs = self.mobilebert( 2025-10-10T01:00:00.4807497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4807574Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4807877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4807954Z layer_outputs = layer_module( 2025-10-10T01:00:00.4808258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4808360Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4808667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4808813Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4809115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4809215Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4809218Z 2025-10-10T01:00:00.4809328Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4809545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4809615Z res = mod(**inputs) 2025-10-10T01:00:00.4809923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4810009Z outputs = self.mobilebert( 2025-10-10T01:00:00.4810312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4810396Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4810697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4810774Z layer_outputs = layer_module( 2025-10-10T01:00:00.4811081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4811184Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4811499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4811636Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4811979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4812113Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4812409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4812516Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4812520Z 2025-10-10T01:00:00.4812629Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4812846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4812916Z res = mod(**inputs) 2025-10-10T01:00:00.4813219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4813293Z outputs = self.mobilebert( 2025-10-10T01:00:00.4813626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4813713Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4814008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4814092Z layer_outputs = layer_module( 2025-10-10T01:00:00.4814390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4814492Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4814804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4814925Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4815232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4815328Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4815332Z 2025-10-10T01:00:00.4815448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4815660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4815730Z res = mod(**inputs) 2025-10-10T01:00:00.4816035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4816111Z outputs = self.mobilebert( 2025-10-10T01:00:00.4816420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4816498Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4816799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4816887Z layer_outputs = layer_module( 2025-10-10T01:00:00.4817184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4817291Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4817597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4817724Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4818032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4818153Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4818157Z 2025-10-10T01:00:00.4818275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4818518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4818613Z res = mod(**inputs) 2025-10-10T01:00:00.4818915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4818987Z outputs = self.mobilebert( 2025-10-10T01:00:00.4819278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4819353Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4819641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4819713Z layer_outputs = layer_module( 2025-10-10T01:00:00.4820008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4820145Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4820450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4820590Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4820892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4820987Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4820991Z 2025-10-10T01:00:00.4821102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4821312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4821390Z res = mod(**inputs) 2025-10-10T01:00:00.4821690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4821778Z outputs = self.mobilebert( 2025-10-10T01:00:00.4822077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4822162Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4822460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4822537Z layer_outputs = layer_module( 2025-10-10T01:00:00.4822844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4822943Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4823247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4823379Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4823685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4823823Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4824125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4824233Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4824236Z 2025-10-10T01:00:00.4824344Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4824560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4824629Z res = mod(**inputs) 2025-10-10T01:00:00.4824927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4825029Z outputs = self.mobilebert( 2025-10-10T01:00:00.4825345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4825433Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4825733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4825811Z layer_outputs = layer_module( 2025-10-10T01:00:00.4826122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4826225Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4826625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4826756Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4827093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4827210Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4827214Z 2025-10-10T01:00:00.4827328Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4827562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4827633Z res = mod(**inputs) 2025-10-10T01:00:00.4827939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4828014Z outputs = self.mobilebert( 2025-10-10T01:00:00.4828312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4828399Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4828698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4828788Z layer_outputs = layer_module( 2025-10-10T01:00:00.4829085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4829193Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4829492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4829614Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4829921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4830042Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4830045Z 2025-10-10T01:00:00.4830163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4830381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4830458Z res = mod(**inputs) 2025-10-10T01:00:00.4830759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4830835Z outputs = self.mobilebert( 2025-10-10T01:00:00.4831146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4831224Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4831731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4831817Z layer_outputs = layer_module( 2025-10-10T01:00:00.4832133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4832299Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4832634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4832781Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4833101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4833201Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4833205Z 2025-10-10T01:00:00.4833329Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4833538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4833615Z res = mod(**inputs) 2025-10-10T01:00:00.4833945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4834069Z outputs = self.mobilebert( 2025-10-10T01:00:00.4834373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4834452Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4834760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4834838Z layer_outputs = layer_module( 2025-10-10T01:00:00.4835151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4835250Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4835557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4835694Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4835996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4836135Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4836436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4836542Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4836546Z 2025-10-10T01:00:00.4836657Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4836870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4836946Z res = mod(**inputs) 2025-10-10T01:00:00.4837256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4837343Z outputs = self.mobilebert( 2025-10-10T01:00:00.4837642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4837726Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4838027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4838104Z layer_outputs = layer_module( 2025-10-10T01:00:00.4838412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4838544Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4838848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4838937Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4838990Z 2025-10-10T01:00:00.4839105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4839339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4839409Z res = mod(**inputs) 2025-10-10T01:00:00.4839713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4839789Z outputs = self.mobilebert( 2025-10-10T01:00:00.4840093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4840171Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4840469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4840553Z layer_outputs = layer_module( 2025-10-10T01:00:00.4840871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4841037Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4841332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4841451Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4841463Z 2025-10-10T01:00:00.4841574Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4841784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4841864Z res = mod(**inputs) 2025-10-10T01:00:00.4842160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4842243Z outputs = self.mobilebert( 2025-10-10T01:00:00.4842543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4842622Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4842923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4842998Z layer_outputs = layer_module( 2025-10-10T01:00:00.4843300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4843467Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4843766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4843876Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4843880Z 2025-10-10T01:00:00.4843992Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4844215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4844284Z res = mod(**inputs) 2025-10-10T01:00:00.4844587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4844664Z outputs = self.mobilebert( 2025-10-10T01:00:00.4844960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4845043Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4845340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4845421Z layer_outputs = layer_module( 2025-10-10T01:00:00.4845717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4845931Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4846234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4846364Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4846669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4846768Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4846772Z 2025-10-10T01:00:00.4846889Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4847100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4847171Z res = mod(**inputs) 2025-10-10T01:00:00.4847490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4847585Z outputs = self.mobilebert( 2025-10-10T01:00:00.4847888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4847964Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4848268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4848345Z layer_outputs = layer_module( 2025-10-10T01:00:00.4848641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4848814Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4849110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4849254Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4849548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4849639Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4849650Z 2025-10-10T01:00:00.4849764Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4849971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4850048Z res = mod(**inputs) 2025-10-10T01:00:00.4850343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4850428Z outputs = self.mobilebert( 2025-10-10T01:00:00.4850724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4850805Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4851109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4851185Z layer_outputs = layer_module( 2025-10-10T01:00:00.4851483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4851638Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4851918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4852049Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4852329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4852496Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4852774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4852875Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4852878Z 2025-10-10T01:00:00.4852981Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4853179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4853251Z res = mod(**inputs) 2025-10-10T01:00:00.4853531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4853610Z outputs = self.mobilebert( 2025-10-10T01:00:00.4853908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4853997Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4854291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4854363Z layer_outputs = layer_module( 2025-10-10T01:00:00.4854653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4854814Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4855109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4855221Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4855508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4855604Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4855609Z 2025-10-10T01:00:00.4855712Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4855921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4855987Z res = mod(**inputs) 2025-10-10T01:00:00.4856287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4856362Z outputs = self.mobilebert( 2025-10-10T01:00:00.4856674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4856759Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4857070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4857159Z layer_outputs = layer_module( 2025-10-10T01:00:00.4857464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4857558Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4857868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4857946Z self_outputs = self.self( 2025-10-10T01:00:00.4858259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4858336Z self.value(value_tensor) 2025-10-10T01:00:00.4858340Z 2025-10-10T01:00:00.4858457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4858669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4858757Z res = mod(**inputs) 2025-10-10T01:00:00.4859084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4859160Z outputs = self.mobilebert( 2025-10-10T01:00:00.4859465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4859542Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4859837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4859921Z layer_outputs = layer_module( 2025-10-10T01:00:00.4860225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4860400Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4860727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4860873Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4861175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4861262Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4861266Z 2025-10-10T01:00:00.4861383Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4861589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4861664Z res = mod(**inputs) 2025-10-10T01:00:00.4861998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4862076Z outputs = self.mobilebert( 2025-10-10T01:00:00.4862392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4862473Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4862790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4862869Z layer_outputs = layer_module( 2025-10-10T01:00:00.4863187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4863361Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4863667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4863796Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4864112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4864217Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4864529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4864631Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4864642Z 2025-10-10T01:00:00.4864755Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4864970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4865049Z res = mod(**inputs) 2025-10-10T01:00:00.4865361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4865446Z outputs = self.mobilebert( 2025-10-10T01:00:00.4865751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4865868Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4866182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4866263Z layer_outputs = layer_module( 2025-10-10T01:00:00.4866648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4866754Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4867063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4867154Z self_outputs = self.self( 2025-10-10T01:00:00.4867464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4867603Z self.query(query_tensor) 2025-10-10T01:00:00.4867607Z 2025-10-10T01:00:00.4867727Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4867952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4868026Z res = mod(**inputs) 2025-10-10T01:00:00.4868346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4868432Z outputs = self.mobilebert( 2025-10-10T01:00:00.4868750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4868838Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4869155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4869232Z layer_outputs = layer_module( 2025-10-10T01:00:00.4869572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4869667Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4869987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4870064Z self_outputs = self.self( 2025-10-10T01:00:00.4870390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4870464Z self.key(key_tensor) 2025-10-10T01:00:00.4870468Z 2025-10-10T01:00:00.4870557Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4870654Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4870766Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4870999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4871074Z res = mod(**inputs) 2025-10-10T01:00:00.4871381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4871466Z outputs = self.mobilebert( 2025-10-10T01:00:00.4871773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4871860Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4872167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4872243Z layer_outputs = layer_module( 2025-10-10T01:00:00.4872567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4872658Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4873001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4873138Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4873436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4873519Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4873522Z 2025-10-10T01:00:00.4873621Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4873815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4873880Z res = mod(**inputs) 2025-10-10T01:00:00.4874153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4874223Z outputs = self.mobilebert( 2025-10-10T01:00:00.4874521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4874602Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4874869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4874943Z layer_outputs = layer_module( 2025-10-10T01:00:00.4875209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4875298Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4875563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4875681Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4875957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4876081Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4876359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4876452Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4876455Z 2025-10-10T01:00:00.4876562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4876752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4876814Z res = mod(**inputs) 2025-10-10T01:00:00.4877090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4877160Z outputs = self.mobilebert( 2025-10-10T01:00:00.4877439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4877512Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4877782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4877860Z layer_outputs = layer_module( 2025-10-10T01:00:00.4878140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4878240Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4878505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4878615Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4878887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4879269Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4879273Z 2025-10-10T01:00:00.4879389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4879583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4879658Z res = mod(**inputs) 2025-10-10T01:00:00.4879929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4879999Z outputs = self.mobilebert( 2025-10-10T01:00:00.4880286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4880359Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4880649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4880755Z layer_outputs = layer_module( 2025-10-10T01:00:00.4881030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4881135Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4881411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4881530Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4881797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4881914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4881918Z 2025-10-10T01:00:00.4882016Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4882204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4882276Z res = mod(**inputs) 2025-10-10T01:00:00.4882542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4882617Z outputs = self.mobilebert( 2025-10-10T01:00:00.4882880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4882950Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4883221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4883290Z layer_outputs = layer_module( 2025-10-10T01:00:00.4883563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4883655Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4883938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4884071Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4884349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4884443Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4884447Z 2025-10-10T01:00:00.4884548Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4884752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4884818Z res = mod(**inputs) 2025-10-10T01:00:00.4885101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4885191Z outputs = self.mobilebert( 2025-10-10T01:00:00.4885501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4885582Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4885854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4885932Z layer_outputs = layer_module( 2025-10-10T01:00:00.4886206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4886299Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4886583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4886708Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4887006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4887147Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4887435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4887530Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4887534Z 2025-10-10T01:00:00.4887638Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4887847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4887913Z res = mod(**inputs) 2025-10-10T01:00:00.4888200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4888273Z outputs = self.mobilebert( 2025-10-10T01:00:00.4888562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4888644Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4888924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4889002Z layer_outputs = layer_module( 2025-10-10T01:00:00.4889294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4889403Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4889702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4889826Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4890139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4890236Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4890240Z 2025-10-10T01:00:00.4890357Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4890574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4890641Z res = mod(**inputs) 2025-10-10T01:00:00.4890930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4891002Z outputs = self.mobilebert( 2025-10-10T01:00:00.4891291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4891366Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4891655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4891774Z layer_outputs = layer_module( 2025-10-10T01:00:00.4892054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4892158Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4892436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4892555Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4892834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4892952Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4892963Z 2025-10-10T01:00:00.4893071Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4893297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4893391Z res = mod(**inputs) 2025-10-10T01:00:00.4893697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4893779Z outputs = self.mobilebert( 2025-10-10T01:00:00.4894082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4894161Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4894470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4894547Z layer_outputs = layer_module( 2025-10-10T01:00:00.4894864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4894964Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4895252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4895386Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4895691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4895789Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4895793Z 2025-10-10T01:00:00.4895903Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4896120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4896191Z res = mod(**inputs) 2025-10-10T01:00:00.4896504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4896594Z outputs = self.mobilebert( 2025-10-10T01:00:00.4896900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4896986Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4897299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4897372Z layer_outputs = layer_module( 2025-10-10T01:00:00.4897667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4897761Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4898058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4898184Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4898515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4898641Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4898919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4899022Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4899025Z 2025-10-10T01:00:00.4899133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4899350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4899422Z res = mod(**inputs) 2025-10-10T01:00:00.4899733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4899810Z outputs = self.mobilebert( 2025-10-10T01:00:00.4900140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4900228Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4900525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4900608Z layer_outputs = layer_module( 2025-10-10T01:00:00.4900911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4901011Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4901315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4901435Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4901745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4901836Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4901840Z 2025-10-10T01:00:00.4901956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4902186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4902255Z res = mod(**inputs) 2025-10-10T01:00:00.4902560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4902635Z outputs = self.mobilebert( 2025-10-10T01:00:00.4902943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4903019Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4903325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4903412Z layer_outputs = layer_module( 2025-10-10T01:00:00.4903718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4903824Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4904120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4904246Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4904548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4904667Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4904671Z 2025-10-10T01:00:00.4904788Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4905033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4905112Z res = mod(**inputs) 2025-10-10T01:00:00.4905420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4905496Z outputs = self.mobilebert( 2025-10-10T01:00:00.4905861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4905940Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4906259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4906407Z layer_outputs = layer_module( 2025-10-10T01:00:00.4906731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4906875Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4907201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4907347Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4907661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4907762Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4907766Z 2025-10-10T01:00:00.4907879Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4908103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4908181Z res = mod(**inputs) 2025-10-10T01:00:00.4908487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4908576Z outputs = self.mobilebert( 2025-10-10T01:00:00.4908885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4908971Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4909284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4909362Z layer_outputs = layer_module( 2025-10-10T01:00:00.4909676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4909780Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4910098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4910235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4910556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4910695Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4911065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4911173Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4911176Z 2025-10-10T01:00:00.4911284Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4911499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4911567Z res = mod(**inputs) 2025-10-10T01:00:00.4911871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4911983Z outputs = self.mobilebert( 2025-10-10T01:00:00.4912295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4912382Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4912674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4912750Z layer_outputs = layer_module( 2025-10-10T01:00:00.4913062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4913193Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4913494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4913585Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4913609Z 2025-10-10T01:00:00.4913746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4913960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4914028Z res = mod(**inputs) 2025-10-10T01:00:00.4914338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4914415Z outputs = self.mobilebert( 2025-10-10T01:00:00.4914729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4914809Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4915126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4915211Z layer_outputs = layer_module( 2025-10-10T01:00:00.4915525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4915669Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4915973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4916103Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4916107Z 2025-10-10T01:00:00.4916218Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4916435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4916514Z res = mod(**inputs) 2025-10-10T01:00:00.4916831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4916917Z outputs = self.mobilebert( 2025-10-10T01:00:00.4917229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4917311Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4917626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4917701Z layer_outputs = layer_module( 2025-10-10T01:00:00.4918011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4918183Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4918506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4918612Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4918616Z 2025-10-10T01:00:00.4918729Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4919002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4919074Z res = mod(**inputs) 2025-10-10T01:00:00.4919391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4919469Z outputs = self.mobilebert( 2025-10-10T01:00:00.4919783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4919863Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4920170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4920255Z layer_outputs = layer_module( 2025-10-10T01:00:00.4920583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4920776Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4921070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4921202Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4921513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4921612Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4921617Z 2025-10-10T01:00:00.4921733Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4921941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4922018Z res = mod(**inputs) 2025-10-10T01:00:00.4922317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4922397Z outputs = self.mobilebert( 2025-10-10T01:00:00.4922701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4922778Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4923080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4923155Z layer_outputs = layer_module( 2025-10-10T01:00:00.4923450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4923620Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4923915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4924058Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4924352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4924450Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4924454Z 2025-10-10T01:00:00.4924564Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4924773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4924850Z res = mod(**inputs) 2025-10-10T01:00:00.4925144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4925225Z outputs = self.mobilebert( 2025-10-10T01:00:00.4925522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4925633Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4925938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4926016Z layer_outputs = layer_module( 2025-10-10T01:00:00.4926325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4926494Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4926808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4926943Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4927248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.4927427Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4927730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4927848Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4927852Z 2025-10-10T01:00:00.4927961Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4928176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4928246Z res = mod(**inputs) 2025-10-10T01:00:00.4928541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4928625Z outputs = self.mobilebert( 2025-10-10T01:00:00.4928922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4929015Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4929318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4929395Z layer_outputs = layer_module( 2025-10-10T01:00:00.4929705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4929879Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4930190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4930310Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4930621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4930717Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4930722Z 2025-10-10T01:00:00.4930835Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4931056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4931128Z res = mod(**inputs) 2025-10-10T01:00:00.4931612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4931706Z outputs = self.mobilebert( 2025-10-10T01:00:00.4932017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4932109Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4932419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4932559Z layer_outputs = layer_module( 2025-10-10T01:00:00.4932892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4933002Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4933305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4933385Z self_outputs = self.self( 2025-10-10T01:00:00.4933700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.4933781Z self.value(value_tensor) 2025-10-10T01:00:00.4933785Z 2025-10-10T01:00:00.4933907Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4934125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4934200Z res = mod(**inputs) 2025-10-10T01:00:00.4934565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4934645Z outputs = self.mobilebert( 2025-10-10T01:00:00.4934958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4935037Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4935352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4935430Z layer_outputs = layer_module( 2025-10-10T01:00:00.4935747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4935931Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4936242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.4936373Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.4936682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.4936771Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.4936783Z 2025-10-10T01:00:00.4936896Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4937115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4937201Z res = mod(**inputs) 2025-10-10T01:00:00.4937514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4937599Z outputs = self.mobilebert( 2025-10-10T01:00:00.4937907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4937989Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4938304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4938383Z layer_outputs = layer_module( 2025-10-10T01:00:00.4938700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.4938875Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.4939177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.4939295Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.4939580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.4939713Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.4939994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4940094Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4940097Z 2025-10-10T01:00:00.4940201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4940398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4940473Z res = mod(**inputs) 2025-10-10T01:00:00.4940750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4940831Z outputs = self.mobilebert( 2025-10-10T01:00:00.4941128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4941259Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4941546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4941618Z layer_outputs = layer_module( 2025-10-10T01:00:00.4941903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4941992Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4942301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4942377Z self_outputs = self.self( 2025-10-10T01:00:00.4942680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.4942769Z self.query(query_tensor) 2025-10-10T01:00:00.4942774Z 2025-10-10T01:00:00.4942886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4943101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4943170Z res = mod(**inputs) 2025-10-10T01:00:00.4943465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4943547Z outputs = self.mobilebert( 2025-10-10T01:00:00.4943843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4943931Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4944237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4944319Z layer_outputs = layer_module( 2025-10-10T01:00:00.4944625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4944718Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4945025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.4945102Z self_outputs = self.self( 2025-10-10T01:00:00.4945404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.4945477Z self.key(key_tensor) 2025-10-10T01:00:00.4945481Z 2025-10-10T01:00:00.4945570Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4945663Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.4945773Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4945987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4946087Z res = mod(**inputs) 2025-10-10T01:00:00.4946471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4946561Z outputs = self.mobilebert( 2025-10-10T01:00:00.4946868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4946953Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4947258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4947342Z layer_outputs = layer_module( 2025-10-10T01:00:00.4947646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4947742Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4948093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4948232Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4948537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.4948625Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4948628Z 2025-10-10T01:00:00.4948741Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4948942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4949008Z res = mod(**inputs) 2025-10-10T01:00:00.4949301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4949373Z outputs = self.mobilebert( 2025-10-10T01:00:00.4949671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4949745Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4950028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4950110Z layer_outputs = layer_module( 2025-10-10T01:00:00.4950390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.4950483Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.4950765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.4950890Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.4951181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.4951313Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4951602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4951696Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4951700Z 2025-10-10T01:00:00.4951813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4952012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4952078Z res = mod(**inputs) 2025-10-10T01:00:00.4952370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4952441Z outputs = self.mobilebert( 2025-10-10T01:00:00.4952733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4952843Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4953131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4953199Z layer_outputs = layer_module( 2025-10-10T01:00:00.4953467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4953567Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4953831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4953947Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4954228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4954326Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4954330Z 2025-10-10T01:00:00.4954435Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4954622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4954692Z res = mod(**inputs) 2025-10-10T01:00:00.4954961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4955035Z outputs = self.mobilebert( 2025-10-10T01:00:00.4955303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4955372Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4955647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4955719Z layer_outputs = layer_module( 2025-10-10T01:00:00.4955995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4956088Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4956356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4956471Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4956738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4956853Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4956858Z 2025-10-10T01:00:00.4956957Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4957154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4957219Z res = mod(**inputs) 2025-10-10T01:00:00.4957484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4957561Z outputs = self.mobilebert( 2025-10-10T01:00:00.4957830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4957910Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4958180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4958248Z layer_outputs = layer_module( 2025-10-10T01:00:00.4958521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4958612Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4958922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4959045Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4959317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4959398Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4959401Z 2025-10-10T01:00:00.4959503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4959705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4959769Z res = mod(**inputs) 2025-10-10T01:00:00.4960049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4960121Z outputs = self.mobilebert( 2025-10-10T01:00:00.4960423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4960501Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4960772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4960850Z layer_outputs = layer_module( 2025-10-10T01:00:00.4961121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4961219Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4961494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4961613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4961889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4962008Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4962280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4962369Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4962372Z 2025-10-10T01:00:00.4962478Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4962665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4962726Z res = mod(**inputs) 2025-10-10T01:00:00.4962997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4963066Z outputs = self.mobilebert( 2025-10-10T01:00:00.4963341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4963412Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4963677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4963751Z layer_outputs = layer_module( 2025-10-10T01:00:00.4964011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4964110Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4964374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4964483Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4964758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4964874Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4964878Z 2025-10-10T01:00:00.4964987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4965174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4965244Z res = mod(**inputs) 2025-10-10T01:00:00.4965518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4965587Z outputs = self.mobilebert( 2025-10-10T01:00:00.4965868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4965939Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4966242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4966333Z layer_outputs = layer_module( 2025-10-10T01:00:00.4966607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4966718Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4966981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4967097Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4967364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4967480Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4967483Z 2025-10-10T01:00:00.4967581Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4967769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4967845Z res = mod(**inputs) 2025-10-10T01:00:00.4968118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4968195Z outputs = self.mobilebert( 2025-10-10T01:00:00.4968469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4968540Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4968818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4968889Z layer_outputs = layer_module( 2025-10-10T01:00:00.4969172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4969270Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4969558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4969695Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4969967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4970058Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4970062Z 2025-10-10T01:00:00.4970163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4970364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4970429Z res = mod(**inputs) 2025-10-10T01:00:00.4970707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4970798Z outputs = self.mobilebert( 2025-10-10T01:00:00.4971094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4971175Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4971445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4971521Z layer_outputs = layer_module( 2025-10-10T01:00:00.4971798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4971892Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4972179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4972303Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4972625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4972749Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4973034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4973128Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4973131Z 2025-10-10T01:00:00.4973233Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4973438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4973503Z res = mod(**inputs) 2025-10-10T01:00:00.4973788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4973861Z outputs = self.mobilebert( 2025-10-10T01:00:00.4974144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4974225Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4974505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4974584Z layer_outputs = layer_module( 2025-10-10T01:00:00.4974866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4974966Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4975251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4975362Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4975650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4975736Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4975740Z 2025-10-10T01:00:00.4975847Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4976043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4976107Z res = mod(**inputs) 2025-10-10T01:00:00.4976394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4976466Z outputs = self.mobilebert( 2025-10-10T01:00:00.4976752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4976824Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4977109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4977215Z layer_outputs = layer_module( 2025-10-10T01:00:00.4977494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4977594Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4977941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.4978066Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.4978371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4978499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4978511Z 2025-10-10T01:00:00.4978614Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4978846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4978920Z res = mod(**inputs) 2025-10-10T01:00:00.4979199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4979279Z outputs = self.mobilebert( 2025-10-10T01:00:00.4979561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4979635Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4979937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4980012Z layer_outputs = layer_module( 2025-10-10T01:00:00.4980325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4980430Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4980728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4980869Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4981165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.4981263Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4981267Z 2025-10-10T01:00:00.4981376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4981597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4981668Z res = mod(**inputs) 2025-10-10T01:00:00.4981982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4982074Z outputs = self.mobilebert( 2025-10-10T01:00:00.4982379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4982466Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4982770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4982848Z layer_outputs = layer_module( 2025-10-10T01:00:00.4983161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.4983264Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.4983585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.4983721Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.4984072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.4984210Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.4984514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4984623Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4984627Z 2025-10-10T01:00:00.4984742Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4984963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4985036Z res = mod(**inputs) 2025-10-10T01:00:00.4985350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4985462Z outputs = self.mobilebert( 2025-10-10T01:00:00.4985771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4985858Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4986162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4986246Z layer_outputs = layer_module( 2025-10-10T01:00:00.4986615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4986754Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4987071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.4987165Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.4987174Z 2025-10-10T01:00:00.4987297Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4987518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4987606Z res = mod(**inputs) 2025-10-10T01:00:00.4987887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4987960Z outputs = self.mobilebert( 2025-10-10T01:00:00.4988249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4988322Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4988607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4988682Z layer_outputs = layer_module( 2025-10-10T01:00:00.4988991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.4989136Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.4989442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.4989575Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.4989579Z 2025-10-10T01:00:00.4989693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4989916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4989988Z res = mod(**inputs) 2025-10-10T01:00:00.4990293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4990382Z outputs = self.mobilebert( 2025-10-10T01:00:00.4990710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4990824Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4991138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4991215Z layer_outputs = layer_module( 2025-10-10T01:00:00.4991534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4991707Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4992028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.4992133Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.4992137Z 2025-10-10T01:00:00.4992257Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4992510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4992583Z res = mod(**inputs) 2025-10-10T01:00:00.4992895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4992972Z outputs = self.mobilebert( 2025-10-10T01:00:00.4993283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4993362Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4993664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4993749Z layer_outputs = layer_module( 2025-10-10T01:00:00.4994053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4994242Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4994547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.4994689Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.4995005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.4995109Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.4995113Z 2025-10-10T01:00:00.4995233Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4995446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4995525Z res = mod(**inputs) 2025-10-10T01:00:00.4995832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4995913Z outputs = self.mobilebert( 2025-10-10T01:00:00.4996230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4996307Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4996607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4996683Z layer_outputs = layer_module( 2025-10-10T01:00:00.4997003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.4997173Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.4997484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.4997662Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.4997962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.4998061Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.4998065Z 2025-10-10T01:00:00.4998175Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.4998393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.4998462Z res = mod(**inputs) 2025-10-10T01:00:00.4998762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.4998847Z outputs = self.mobilebert( 2025-10-10T01:00:00.4999161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.4999277Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.4999560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.4999631Z layer_outputs = layer_module( 2025-10-10T01:00:00.4999920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5000075Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5000364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5000486Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5000775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5000902Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5001185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5001286Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5001289Z 2025-10-10T01:00:00.5001395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5001599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5001664Z res = mod(**inputs) 2025-10-10T01:00:00.5001950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5002030Z outputs = self.mobilebert( 2025-10-10T01:00:00.5002313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5002403Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5002702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5002787Z layer_outputs = layer_module( 2025-10-10T01:00:00.5003092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5003262Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5003572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5003689Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5003994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5004116Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5004120Z 2025-10-10T01:00:00.5004249Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5004448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5004514Z res = mod(**inputs) 2025-10-10T01:00:00.5004801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5004874Z outputs = self.mobilebert( 2025-10-10T01:00:00.5005161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5005234Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5005526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5005629Z layer_outputs = layer_module( 2025-10-10T01:00:00.5005945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5006049Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5006345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5006421Z self_outputs = self.self( 2025-10-10T01:00:00.5006726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.5006803Z self.value(value_tensor) 2025-10-10T01:00:00.5006807Z 2025-10-10T01:00:00.5006924Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5007135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5007213Z res = mod(**inputs) 2025-10-10T01:00:00.5007516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5007589Z outputs = self.mobilebert( 2025-10-10T01:00:00.5007890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5007968Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5008269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5008346Z layer_outputs = layer_module( 2025-10-10T01:00:00.5008641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5008816Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5009116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.5009243Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.5009540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5009634Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5009638Z 2025-10-10T01:00:00.5009746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5009954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5010030Z res = mod(**inputs) 2025-10-10T01:00:00.5010329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5010409Z outputs = self.mobilebert( 2025-10-10T01:00:00.5010710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5010826Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5011130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5011205Z layer_outputs = layer_module( 2025-10-10T01:00:00.5011508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5011676Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5011980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5012098Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5012412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.5012531Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.5012830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5012936Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5012939Z 2025-10-10T01:00:00.5013049Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5013266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5013335Z res = mod(**inputs) 2025-10-10T01:00:00.5013633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5013716Z outputs = self.mobilebert( 2025-10-10T01:00:00.5014014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5014105Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5014407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5014483Z layer_outputs = layer_module( 2025-10-10T01:00:00.5014789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5014881Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5015187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5015264Z self_outputs = self.self( 2025-10-10T01:00:00.5015570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.5015649Z self.query(query_tensor) 2025-10-10T01:00:00.5015657Z 2025-10-10T01:00:00.5015769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5015988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5016058Z res = mod(**inputs) 2025-10-10T01:00:00.5016363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5016438Z outputs = self.mobilebert( 2025-10-10T01:00:00.5016735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5016821Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5017118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5017203Z layer_outputs = layer_module( 2025-10-10T01:00:00.5017545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5017645Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5017957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5018036Z self_outputs = self.self( 2025-10-10T01:00:00.5018357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.5018433Z self.key(key_tensor) 2025-10-10T01:00:00.5018437Z 2025-10-10T01:00:00.5018534Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5018625Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5018740Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5018970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5019076Z res = mod(**inputs) 2025-10-10T01:00:00.5019382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5019458Z outputs = self.mobilebert( 2025-10-10T01:00:00.5019753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5019839Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5020144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5020229Z layer_outputs = layer_module( 2025-10-10T01:00:00.5020532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5020631Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5020934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5021069Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5021373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.5021464Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5021468Z 2025-10-10T01:00:00.5021585Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5021795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5021863Z res = mod(**inputs) 2025-10-10T01:00:00.5022176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5022252Z outputs = self.mobilebert( 2025-10-10T01:00:00.5022561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5022639Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5022942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5023017Z layer_outputs = layer_module( 2025-10-10T01:00:00.5023323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5023423Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5023727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5023868Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5024175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.5024350Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5024661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5024761Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5024765Z 2025-10-10T01:00:00.5024884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5025096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5025174Z res = mod(**inputs) 2025-10-10T01:00:00.5025470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5025545Z outputs = self.mobilebert( 2025-10-10T01:00:00.5025875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5025976Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5026288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5026431Z layer_outputs = layer_module( 2025-10-10T01:00:00.5026748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5026864Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5027172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5027307Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5027616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5027724Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5027729Z 2025-10-10T01:00:00.5027845Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5028068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5028149Z res = mod(**inputs) 2025-10-10T01:00:00.5028462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5028548Z outputs = self.mobilebert( 2025-10-10T01:00:00.5028848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5028928Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5029237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5029321Z layer_outputs = layer_module( 2025-10-10T01:00:00.5029640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5029747Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5030066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5030193Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5030499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5030634Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5030639Z 2025-10-10T01:00:00.5030753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5030981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5031080Z res = mod(**inputs) 2025-10-10T01:00:00.5031408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5031709Z outputs = self.mobilebert( 2025-10-10T01:00:00.5032023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5032112Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5032419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5032508Z layer_outputs = layer_module( 2025-10-10T01:00:00.5032811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5032916Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5033300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5033443Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5033761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5033857Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5033862Z 2025-10-10T01:00:00.5033984Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5034203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5034276Z res = mod(**inputs) 2025-10-10T01:00:00.5034596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5034676Z outputs = self.mobilebert( 2025-10-10T01:00:00.5035003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5035085Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5035396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5035484Z layer_outputs = layer_module( 2025-10-10T01:00:00.5035794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5035912Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5036224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5036362Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5036687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5036824Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5037140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5037244Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5037248Z 2025-10-10T01:00:00.5037368Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5037588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5037659Z res = mod(**inputs) 2025-10-10T01:00:00.5037987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5038065Z outputs = self.mobilebert( 2025-10-10T01:00:00.5038445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5038528Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5038839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5038918Z layer_outputs = layer_module( 2025-10-10T01:00:00.5039236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5039348Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5039662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5039793Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5040111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5040224Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5040228Z 2025-10-10T01:00:00.5040337Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5040532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5040603Z res = mod(**inputs) 2025-10-10T01:00:00.5040883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5040964Z outputs = self.mobilebert( 2025-10-10T01:00:00.5041245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5041321Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5041611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5041686Z layer_outputs = layer_module( 2025-10-10T01:00:00.5041978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5042073Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5042356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5042479Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5042771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5042887Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5042891Z 2025-10-10T01:00:00.5042993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5043196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5043265Z res = mod(**inputs) 2025-10-10T01:00:00.5043540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5043617Z outputs = self.mobilebert( 2025-10-10T01:00:00.5043893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5043971Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5044248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5044320Z layer_outputs = layer_module( 2025-10-10T01:00:00.5044607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5044723Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5045027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5045165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5045442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5045526Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5045529Z 2025-10-10T01:00:00.5045630Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5045831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5045894Z res = mod(**inputs) 2025-10-10T01:00:00.5046172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5046322Z outputs = self.mobilebert( 2025-10-10T01:00:00.5046618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5046702Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5046995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5047079Z layer_outputs = layer_module( 2025-10-10T01:00:00.5047374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5047481Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5047777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5047909Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5048228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5048348Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5048626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5048718Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5048721Z 2025-10-10T01:00:00.5048828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5049021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5049085Z res = mod(**inputs) 2025-10-10T01:00:00.5049365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5049434Z outputs = self.mobilebert( 2025-10-10T01:00:00.5049720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5049792Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5050063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5050141Z layer_outputs = layer_module( 2025-10-10T01:00:00.5050412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5050513Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5050785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5050897Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5051175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5051301Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5051304Z 2025-10-10T01:00:00.5051415Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5051608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5051680Z res = mod(**inputs) 2025-10-10T01:00:00.5051956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5052028Z outputs = self.mobilebert( 2025-10-10T01:00:00.5052313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5052387Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5052707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5052810Z layer_outputs = layer_module( 2025-10-10T01:00:00.5053107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5053215Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5053511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5053638Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5053939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5054056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5054060Z 2025-10-10T01:00:00.5054160Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5054359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5054432Z res = mod(**inputs) 2025-10-10T01:00:00.5054703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5054780Z outputs = self.mobilebert( 2025-10-10T01:00:00.5055053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5055124Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5055401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5055471Z layer_outputs = layer_module( 2025-10-10T01:00:00.5055763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5055868Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5056172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5056307Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5056603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5056700Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5056704Z 2025-10-10T01:00:00.5056813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5057031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5057100Z res = mod(**inputs) 2025-10-10T01:00:00.5057414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5057510Z outputs = self.mobilebert( 2025-10-10T01:00:00.5057827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5057914Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5058210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5058294Z layer_outputs = layer_module( 2025-10-10T01:00:00.5058601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5058703Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5059017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5059150Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5059490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5059622Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5059927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5060027Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5060030Z 2025-10-10T01:00:00.5060139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5060353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5060422Z res = mod(**inputs) 2025-10-10T01:00:00.5060734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5060812Z outputs = self.mobilebert( 2025-10-10T01:00:00.5061112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5061197Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5061492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5061572Z layer_outputs = layer_module( 2025-10-10T01:00:00.5061872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5062009Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5062315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5062406Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5062412Z 2025-10-10T01:00:00.5062533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5062742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5062819Z res = mod(**inputs) 2025-10-10T01:00:00.5063115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5063190Z outputs = self.mobilebert( 2025-10-10T01:00:00.5063494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5063571Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5063877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5063953Z layer_outputs = layer_module( 2025-10-10T01:00:00.5064255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5064417Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5064721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5064851Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5064855Z 2025-10-10T01:00:00.5064970Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5065189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5065261Z res = mod(**inputs) 2025-10-10T01:00:00.5065575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5065660Z outputs = self.mobilebert( 2025-10-10T01:00:00.5065996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5066104Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5066479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5066572Z layer_outputs = layer_module( 2025-10-10T01:00:00.5066879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5067054Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5067368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.5067475Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.5067481Z 2025-10-10T01:00:00.5067602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5067828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5067901Z res = mod(**inputs) 2025-10-10T01:00:00.5068220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5068298Z outputs = self.mobilebert( 2025-10-10T01:00:00.5068603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5068682Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5068987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5069065Z layer_outputs = layer_module( 2025-10-10T01:00:00.5069361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5069544Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5069843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.5069986Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.5070282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5070392Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5070396Z 2025-10-10T01:00:00.5070507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5070715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5070792Z res = mod(**inputs) 2025-10-10T01:00:00.5071099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5071224Z outputs = self.mobilebert( 2025-10-10T01:00:00.5071526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5071604Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5071908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5071984Z layer_outputs = layer_module( 2025-10-10T01:00:00.5072289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5072454Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5072758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5072928Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5073225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.5073323Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5073328Z 2025-10-10T01:00:00.5073437Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5073658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5073727Z res = mod(**inputs) 2025-10-10T01:00:00.5074023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5074106Z outputs = self.mobilebert( 2025-10-10T01:00:00.5074405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5074494Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5074788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5074863Z layer_outputs = layer_module( 2025-10-10T01:00:00.5075135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5075287Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5075569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5075690Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5075969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5076097Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5076374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5076474Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5076477Z 2025-10-10T01:00:00.5076578Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5076780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5076842Z res = mod(**inputs) 2025-10-10T01:00:00.5077124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5077195Z outputs = self.mobilebert( 2025-10-10T01:00:00.5077476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5077579Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5077874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5077955Z layer_outputs = layer_module( 2025-10-10T01:00:00.5078239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5078402Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5078697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5078812Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5079103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5079222Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5079226Z 2025-10-10T01:00:00.5079337Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5079539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5091692Z res = mod(**inputs) 2025-10-10T01:00:00.5092101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5092191Z outputs = self.mobilebert( 2025-10-10T01:00:00.5092521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5092607Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5092937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5093027Z layer_outputs = layer_module( 2025-10-10T01:00:00.5093356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5093460Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5093754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5093839Z self_outputs = self.self( 2025-10-10T01:00:00.5094130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.5094215Z self.value(value_tensor) 2025-10-10T01:00:00.5094220Z 2025-10-10T01:00:00.5094339Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5094557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5094637Z res = mod(**inputs) 2025-10-10T01:00:00.5094953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5095040Z outputs = self.mobilebert( 2025-10-10T01:00:00.5095355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5095437Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5095765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5095842Z layer_outputs = layer_module( 2025-10-10T01:00:00.5096158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5096338Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5096668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.5096908Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.5097218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5097319Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5097323Z 2025-10-10T01:00:00.5097442Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5097673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5097746Z res = mod(**inputs) 2025-10-10T01:00:00.5098057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5098146Z outputs = self.mobilebert( 2025-10-10T01:00:00.5098485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5098609Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5098930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5099017Z layer_outputs = layer_module( 2025-10-10T01:00:00.5099325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5099500Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5099820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5099949Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5100269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.5100373Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.5100702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5100813Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5100817Z 2025-10-10T01:00:00.5100943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5101165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5101240Z res = mod(**inputs) 2025-10-10T01:00:00.5101559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5101641Z outputs = self.mobilebert( 2025-10-10T01:00:00.5101964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5102052Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5102369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5102450Z layer_outputs = layer_module( 2025-10-10T01:00:00.5102760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5102870Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5103188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5103275Z self_outputs = self.self( 2025-10-10T01:00:00.5103593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.5103692Z self.query(query_tensor) 2025-10-10T01:00:00.5103707Z 2025-10-10T01:00:00.5103852Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5104071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5104152Z res = mod(**inputs) 2025-10-10T01:00:00.5104459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5104547Z outputs = self.mobilebert( 2025-10-10T01:00:00.5104863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5104946Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5105259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5105340Z layer_outputs = layer_module( 2025-10-10T01:00:00.5105692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5105791Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5106100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5106190Z self_outputs = self.self( 2025-10-10T01:00:00.5106746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.5106840Z self.key(key_tensor) 2025-10-10T01:00:00.5106844Z 2025-10-10T01:00:00.5106939Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5107037Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5107156Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5107374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5107463Z res = mod(**inputs) 2025-10-10T01:00:00.5107776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5107861Z outputs = self.mobilebert( 2025-10-10T01:00:00.5108163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5108245Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5108559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5108637Z layer_outputs = layer_module( 2025-10-10T01:00:00.5108951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5109048Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5109363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5109517Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5109824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.5109929Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5109933Z 2025-10-10T01:00:00.5110049Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5110277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5110350Z res = mod(**inputs) 2025-10-10T01:00:00.5110670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5110758Z outputs = self.mobilebert( 2025-10-10T01:00:00.5111112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5111203Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5111511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5111590Z layer_outputs = layer_module( 2025-10-10T01:00:00.5111916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5112010Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5112332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5112470Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5112802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.5112969Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5113277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5113392Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5113396Z 2025-10-10T01:00:00.5113511Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5113738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5113810Z res = mod(**inputs) 2025-10-10T01:00:00.5114125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5114207Z outputs = self.mobilebert( 2025-10-10T01:00:00.5114520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5114613Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5114919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5115008Z layer_outputs = layer_module( 2025-10-10T01:00:00.5115314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5115424Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5115742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5115868Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5116184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5116284Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5116290Z 2025-10-10T01:00:00.5116410Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5116631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5116704Z res = mod(**inputs) 2025-10-10T01:00:00.5117020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5117100Z outputs = self.mobilebert( 2025-10-10T01:00:00.5117413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5117494Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5117800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5117910Z layer_outputs = layer_module( 2025-10-10T01:00:00.5118234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5118352Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5118660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5118796Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5119105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5119234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5119238Z 2025-10-10T01:00:00.5119358Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5119579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5119696Z res = mod(**inputs) 2025-10-10T01:00:00.5120010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5120090Z outputs = self.mobilebert( 2025-10-10T01:00:00.5120414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5120495Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5120811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5120889Z layer_outputs = layer_module( 2025-10-10T01:00:00.5121207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5121314Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5121629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5121782Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5122100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5122201Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5122204Z 2025-10-10T01:00:00.5122315Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5122530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5122609Z res = mod(**inputs) 2025-10-10T01:00:00.5122911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5122997Z outputs = self.mobilebert( 2025-10-10T01:00:00.5123304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5123390Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5123689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5123768Z layer_outputs = layer_module( 2025-10-10T01:00:00.5124078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5124179Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5124497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5124636Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5125006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5125154Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5125458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5125568Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5125571Z 2025-10-10T01:00:00.5125682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5125901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5125971Z res = mod(**inputs) 2025-10-10T01:00:00.5126274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5126360Z outputs = self.mobilebert( 2025-10-10T01:00:00.5126691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5126798Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5127104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5127182Z layer_outputs = layer_module( 2025-10-10T01:00:00.5127508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5127612Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5127927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5128053Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5128379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5128477Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5128481Z 2025-10-10T01:00:00.5128591Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5128814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5128887Z res = mod(**inputs) 2025-10-10T01:00:00.5129199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5129276Z outputs = self.mobilebert( 2025-10-10T01:00:00.5129582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5129669Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5129977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5130067Z layer_outputs = layer_module( 2025-10-10T01:00:00.5130366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5130477Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5130776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5130898Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5131219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5131342Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5131346Z 2025-10-10T01:00:00.5131666Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5131897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5132080Z res = mod(**inputs) 2025-10-10T01:00:00.5132388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5132466Z outputs = self.mobilebert( 2025-10-10T01:00:00.5132772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5132851Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5133157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5133233Z layer_outputs = layer_module( 2025-10-10T01:00:00.5133530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5133645Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5133995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5134142Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5134441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5134539Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5134544Z 2025-10-10T01:00:00.5134653Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5134864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5134941Z res = mod(**inputs) 2025-10-10T01:00:00.5135239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5135324Z outputs = self.mobilebert( 2025-10-10T01:00:00.5135621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5135700Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5136006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5136083Z layer_outputs = layer_module( 2025-10-10T01:00:00.5136387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5136488Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5136795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5136929Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5137232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5137372Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5137669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5137778Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5137781Z 2025-10-10T01:00:00.5137891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5138102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5138181Z res = mod(**inputs) 2025-10-10T01:00:00.5138483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5138630Z outputs = self.mobilebert( 2025-10-10T01:00:00.5139012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5139104Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5139445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5139564Z layer_outputs = layer_module( 2025-10-10T01:00:00.5139894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5140012Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5140315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5140449Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5140770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5140886Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5140890Z 2025-10-10T01:00:00.5141032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5141293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5141379Z res = mod(**inputs) 2025-10-10T01:00:00.5141687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5141778Z outputs = self.mobilebert( 2025-10-10T01:00:00.5142086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5142174Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5142488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5142573Z layer_outputs = layer_module( 2025-10-10T01:00:00.5142891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5142997Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5143312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5143439Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5143753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5143878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5143882Z 2025-10-10T01:00:00.5143999Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5144234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5144309Z res = mod(**inputs) 2025-10-10T01:00:00.5144628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5144708Z outputs = self.mobilebert( 2025-10-10T01:00:00.5145023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5145103Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5145411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5145499Z layer_outputs = layer_module( 2025-10-10T01:00:00.5145808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5145950Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5146275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5146482Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5146819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5146916Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5146920Z 2025-10-10T01:00:00.5147043Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5147263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5147345Z res = mod(**inputs) 2025-10-10T01:00:00.5147656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5147777Z outputs = self.mobilebert( 2025-10-10T01:00:00.5148076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5148151Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5148441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5148515Z layer_outputs = layer_module( 2025-10-10T01:00:00.5148810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5148920Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5149203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5149339Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5149630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5149763Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5150051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5150149Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5150153Z 2025-10-10T01:00:00.5150269Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5150472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5150545Z res = mod(**inputs) 2025-10-10T01:00:00.5150829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5150906Z outputs = self.mobilebert( 2025-10-10T01:00:00.5151206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5151280Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5151570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5151643Z layer_outputs = layer_module( 2025-10-10T01:00:00.5151934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5152063Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5152349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5152445Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5153129Z 2025-10-10T01:00:00.5153247Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5153481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5153551Z res = mod(**inputs) 2025-10-10T01:00:00.5153840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5153931Z outputs = self.mobilebert( 2025-10-10T01:00:00.5154233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5154318Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5154619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5154706Z layer_outputs = layer_module( 2025-10-10T01:00:00.5155022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5155189Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5155500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5155624Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5155628Z 2025-10-10T01:00:00.5155750Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5155967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5156039Z res = mod(**inputs) 2025-10-10T01:00:00.5156361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5156434Z outputs = self.mobilebert( 2025-10-10T01:00:00.5156729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5156809Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5157102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5157177Z layer_outputs = layer_module( 2025-10-10T01:00:00.5157527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5157713Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5158015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.5158127Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.5158131Z 2025-10-10T01:00:00.5158243Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5158473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5158544Z res = mod(**inputs) 2025-10-10T01:00:00.5158845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5158932Z outputs = self.mobilebert( 2025-10-10T01:00:00.5159231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5159318Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5159618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5159694Z layer_outputs = layer_module( 2025-10-10T01:00:00.5160000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5160210Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5160517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.5160652Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.5160960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5161063Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5161067Z 2025-10-10T01:00:00.5161179Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5161399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5161469Z res = mod(**inputs) 2025-10-10T01:00:00.5161794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5161891Z outputs = self.mobilebert( 2025-10-10T01:00:00.5162191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5162276Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5162577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5162663Z layer_outputs = layer_module( 2025-10-10T01:00:00.5162962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5163142Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5163445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5163591Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5163903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.5163995Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5164000Z 2025-10-10T01:00:00.5164126Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5164344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5164416Z res = mod(**inputs) 2025-10-10T01:00:00.5164726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5164808Z outputs = self.mobilebert( 2025-10-10T01:00:00.5165122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5165207Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5165521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5165600Z layer_outputs = layer_module( 2025-10-10T01:00:00.5165901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5166078Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5166377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5166518Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5166818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5166973Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5167323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5167426Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5167431Z 2025-10-10T01:00:00.5167553Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5167763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5167840Z res = mod(**inputs) 2025-10-10T01:00:00.5168144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5168221Z outputs = self.mobilebert( 2025-10-10T01:00:00.5168528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5168647Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5168953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5169039Z layer_outputs = layer_module( 2025-10-10T01:00:00.5169345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5169519Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5169806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5169927Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5170212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5170303Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5170306Z 2025-10-10T01:00:00.5170423Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5170625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5170700Z res = mod(**inputs) 2025-10-10T01:00:00.5170986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5171059Z outputs = self.mobilebert( 2025-10-10T01:00:00.5171367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5171444Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5171761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5171836Z layer_outputs = layer_module( 2025-10-10T01:00:00.5172130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5172223Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5172508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5172590Z self_outputs = self.self( 2025-10-10T01:00:00.5172877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.5172961Z self.value(value_tensor) 2025-10-10T01:00:00.5172964Z 2025-10-10T01:00:00.5173068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5173267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5173344Z res = mod(**inputs) 2025-10-10T01:00:00.5173664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5173745Z outputs = self.mobilebert( 2025-10-10T01:00:00.5174032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5174112Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5174396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5174468Z layer_outputs = layer_module( 2025-10-10T01:00:00.5174761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5174926Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5175235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.5175366Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.5175656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5175752Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5175756Z 2025-10-10T01:00:00.5175865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5176083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5176155Z res = mod(**inputs) 2025-10-10T01:00:00.5176465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5176539Z outputs = self.mobilebert( 2025-10-10T01:00:00.5176841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5176937Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5177243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5177330Z layer_outputs = layer_module( 2025-10-10T01:00:00.5177643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5177814Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5178122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5178239Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5178549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.5178647Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.5178954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5179049Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5179053Z 2025-10-10T01:00:00.5179158Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5179365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5179430Z res = mod(**inputs) 2025-10-10T01:00:00.5179715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5179786Z outputs = self.mobilebert( 2025-10-10T01:00:00.5180067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5180186Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5180468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5180547Z layer_outputs = layer_module( 2025-10-10T01:00:00.5180838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5180939Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5181239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5181315Z self_outputs = self.self( 2025-10-10T01:00:00.5181624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.5181704Z self.query(query_tensor) 2025-10-10T01:00:00.5181751Z 2025-10-10T01:00:00.5181870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5182080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5182149Z res = mod(**inputs) 2025-10-10T01:00:00.5182461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5182536Z outputs = self.mobilebert( 2025-10-10T01:00:00.5182845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5182923Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5183231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5183305Z layer_outputs = layer_module( 2025-10-10T01:00:00.5183609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5183710Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5184011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5184095Z self_outputs = self.self( 2025-10-10T01:00:00.5184395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.5184467Z self.key(key_tensor) 2025-10-10T01:00:00.5184478Z 2025-10-10T01:00:00.5184567Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5184652Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5184772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5184983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5185059Z res = mod(**inputs) 2025-10-10T01:00:00.5185369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5185444Z outputs = self.mobilebert( 2025-10-10T01:00:00.5185753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5185833Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5186148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5186226Z layer_outputs = layer_module( 2025-10-10T01:00:00.5186630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5186746Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5187110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5187263Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5187578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.5187671Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5187683Z 2025-10-10T01:00:00.5187794Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5188007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5188090Z res = mod(**inputs) 2025-10-10T01:00:00.5188387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5188470Z outputs = self.mobilebert( 2025-10-10T01:00:00.5188787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5188882Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5189187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5189263Z layer_outputs = layer_module( 2025-10-10T01:00:00.5189576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5189668Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5189975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5190119Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5190430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.5190579Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5190888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5190996Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5190999Z 2025-10-10T01:00:00.5191111Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5191324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5191402Z res = mod(**inputs) 2025-10-10T01:00:00.5191706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5191789Z outputs = self.mobilebert( 2025-10-10T01:00:00.5192095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5192184Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5192493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5192569Z layer_outputs = layer_module( 2025-10-10T01:00:00.5192887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5192991Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5193302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5193424Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5193737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5193871Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5193891Z 2025-10-10T01:00:00.5194009Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5194230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5194299Z res = mod(**inputs) 2025-10-10T01:00:00.5194607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5194681Z outputs = self.mobilebert( 2025-10-10T01:00:00.5194983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5195069Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5195402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5195524Z layer_outputs = layer_module( 2025-10-10T01:00:00.5195822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5195927Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5196231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5196353Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5196663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5196784Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5196788Z 2025-10-10T01:00:00.5196907Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5197120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5197196Z res = mod(**inputs) 2025-10-10T01:00:00.5197506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5197582Z outputs = self.mobilebert( 2025-10-10T01:00:00.5197887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5197965Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5198261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5198345Z layer_outputs = layer_module( 2025-10-10T01:00:00.5198648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5198756Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5199063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5199205Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5199501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5199592Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5199596Z 2025-10-10T01:00:00.5199713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5199931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5200010Z res = mod(**inputs) 2025-10-10T01:00:00.5200314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5200390Z outputs = self.mobilebert( 2025-10-10T01:00:00.5200755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5200836Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5201147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5201228Z layer_outputs = layer_module( 2025-10-10T01:00:00.5201541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5201646Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5201956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5202102Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5202430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5202593Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5202903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5203015Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5203025Z 2025-10-10T01:00:00.5203134Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5203349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5203425Z res = mod(**inputs) 2025-10-10T01:00:00.5203727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5203810Z outputs = self.mobilebert( 2025-10-10T01:00:00.5204111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5204192Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5204514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5204592Z layer_outputs = layer_module( 2025-10-10T01:00:00.5204912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5205029Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5205326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5205458Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5205769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5205876Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5205880Z 2025-10-10T01:00:00.5205993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5206229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5206299Z res = mod(**inputs) 2025-10-10T01:00:00.5206598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5206682Z outputs = self.mobilebert( 2025-10-10T01:00:00.5206991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5207078Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5207387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5207497Z layer_outputs = layer_module( 2025-10-10T01:00:00.5207840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5207946Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5208267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5208393Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5208715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5208851Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5208855Z 2025-10-10T01:00:00.5208965Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5209202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5209290Z res = mod(**inputs) 2025-10-10T01:00:00.5209600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5209676Z outputs = self.mobilebert( 2025-10-10T01:00:00.5209977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5210065Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5210363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5210446Z layer_outputs = layer_module( 2025-10-10T01:00:00.5210743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5210853Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5211156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5211288Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5211592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5211683Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5211686Z 2025-10-10T01:00:00.5211804Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5212015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5212092Z res = mod(**inputs) 2025-10-10T01:00:00.5212387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5212466Z outputs = self.mobilebert( 2025-10-10T01:00:00.5212777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5212855Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5213160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5213236Z layer_outputs = layer_module( 2025-10-10T01:00:00.5213534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5213644Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5213942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5214080Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5214421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5214563Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5214860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5214960Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5214963Z 2025-10-10T01:00:00.5215081Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5215292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5215369Z res = mod(**inputs) 2025-10-10T01:00:00.5215667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5215743Z outputs = self.mobilebert( 2025-10-10T01:00:00.5216083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5216163Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5216476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5216551Z layer_outputs = layer_module( 2025-10-10T01:00:00.5216872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5216978Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5217297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5217426Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5217730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5217834Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5217838Z 2025-10-10T01:00:00.5217949Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5218164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5218240Z res = mod(**inputs) 2025-10-10T01:00:00.5218547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5218631Z outputs = self.mobilebert( 2025-10-10T01:00:00.5218948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5219035Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5219369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5219453Z layer_outputs = layer_module( 2025-10-10T01:00:00.5219776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5219881Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5220198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5220323Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5220635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5220768Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5220772Z 2025-10-10T01:00:00.5220887Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5221142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5221231Z res = mod(**inputs) 2025-10-10T01:00:00.5221547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5221628Z outputs = self.mobilebert( 2025-10-10T01:00:00.5221933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5222019Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5222325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5222410Z layer_outputs = layer_module( 2025-10-10T01:00:00.5222715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5222858Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5223174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5223311Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5223625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5223718Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5223723Z 2025-10-10T01:00:00.5223844Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5224057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5224130Z res = mod(**inputs) 2025-10-10T01:00:00.5224444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5224529Z outputs = self.mobilebert( 2025-10-10T01:00:00.5224844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5224923Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5225230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5225316Z layer_outputs = layer_module( 2025-10-10T01:00:00.5225621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5225731Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5226036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5226179Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5226578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5226726Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5227048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5227159Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5227163Z 2025-10-10T01:00:00.5227286Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5227503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5227584Z res = mod(**inputs) 2025-10-10T01:00:00.5227893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5228010Z outputs = self.mobilebert( 2025-10-10T01:00:00.5228370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5228455Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5228769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5228847Z layer_outputs = layer_module( 2025-10-10T01:00:00.5229155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5229299Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5229614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5229709Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5229715Z 2025-10-10T01:00:00.5229873Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5230094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5230174Z res = mod(**inputs) 2025-10-10T01:00:00.5230481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5230557Z outputs = self.mobilebert( 2025-10-10T01:00:00.5230880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5230960Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5231266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5231351Z layer_outputs = layer_module( 2025-10-10T01:00:00.5231849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5232006Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5232323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5232452Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5232466Z 2025-10-10T01:00:00.5232583Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5232799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5232879Z res = mod(**inputs) 2025-10-10T01:00:00.5233190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5233319Z outputs = self.mobilebert( 2025-10-10T01:00:00.5233644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5233753Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5234081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5234175Z layer_outputs = layer_module( 2025-10-10T01:00:00.5234482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5234656Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5234969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.5235077Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.5235081Z 2025-10-10T01:00:00.5235203Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5235520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5235600Z res = mod(**inputs) 2025-10-10T01:00:00.5235903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5235982Z outputs = self.mobilebert( 2025-10-10T01:00:00.5236297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5236377Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5236689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5236768Z layer_outputs = layer_module( 2025-10-10T01:00:00.5237079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5237315Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5237624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.5237768Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.5238073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5238181Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5238185Z 2025-10-10T01:00:00.5238299Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5238514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5238592Z res = mod(**inputs) 2025-10-10T01:00:00.5238899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5238989Z outputs = self.mobilebert( 2025-10-10T01:00:00.5239291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5239370Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5239706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5239781Z layer_outputs = layer_module( 2025-10-10T01:00:00.5240081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5240248Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5240542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5240669Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5240944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.5241037Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5241040Z 2025-10-10T01:00:00.5241142Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5241344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5241409Z res = mod(**inputs) 2025-10-10T01:00:00.5241687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5241765Z outputs = self.mobilebert( 2025-10-10T01:00:00.5242042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5242148Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5242459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5242542Z layer_outputs = layer_module( 2025-10-10T01:00:00.5242844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5243014Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5243323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5243468Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5243763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5243925Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5244221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5244318Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5244322Z 2025-10-10T01:00:00.5244426Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5244637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5244706Z res = mod(**inputs) 2025-10-10T01:00:00.5244998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5245076Z outputs = self.mobilebert( 2025-10-10T01:00:00.5245366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5245456Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5245759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5245842Z layer_outputs = layer_module( 2025-10-10T01:00:00.5246141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5246318Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5246619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5246739Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5247054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5247147Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5247151Z 2025-10-10T01:00:00.5247268Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5247479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5247555Z res = mod(**inputs) 2025-10-10T01:00:00.5247851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5247931Z outputs = self.mobilebert( 2025-10-10T01:00:00.5248242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5248322Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5248626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5248725Z layer_outputs = layer_module( 2025-10-10T01:00:00.5249041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5249147Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5249442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5249523Z self_outputs = self.self( 2025-10-10T01:00:00.5249827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.5249910Z self.value(value_tensor) 2025-10-10T01:00:00.5249914Z 2025-10-10T01:00:00.5250024Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5250236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5250314Z res = mod(**inputs) 2025-10-10T01:00:00.5250649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5250733Z outputs = self.mobilebert( 2025-10-10T01:00:00.5251037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5251115Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5251421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5251497Z layer_outputs = layer_module( 2025-10-10T01:00:00.5251802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5251972Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5252287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.5252405Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.5252697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5252793Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5252797Z 2025-10-10T01:00:00.5252909Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5253124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5253192Z res = mod(**inputs) 2025-10-10T01:00:00.5253497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5253576Z outputs = self.mobilebert( 2025-10-10T01:00:00.5253861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5253944Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5254231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5254312Z layer_outputs = layer_module( 2025-10-10T01:00:00.5254594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5254754Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5255043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5255153Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5255443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.5255573Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.5255860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5255965Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5255969Z 2025-10-10T01:00:00.5256078Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5256297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5256368Z res = mod(**inputs) 2025-10-10T01:00:00.5256667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5256743Z outputs = self.mobilebert( 2025-10-10T01:00:00.5257059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5257171Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5257473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5257560Z layer_outputs = layer_module( 2025-10-10T01:00:00.5257860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5257955Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5258267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5258345Z self_outputs = self.self( 2025-10-10T01:00:00.5258652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.5258728Z self.query(query_tensor) 2025-10-10T01:00:00.5258734Z 2025-10-10T01:00:00.5258856Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5259068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5259138Z res = mod(**inputs) 2025-10-10T01:00:00.5259449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5259527Z outputs = self.mobilebert( 2025-10-10T01:00:00.5259834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5259911Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5260214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5260297Z layer_outputs = layer_module( 2025-10-10T01:00:00.5260606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5260710Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5261011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5261097Z self_outputs = self.self( 2025-10-10T01:00:00.5261397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.5261472Z self.key(key_tensor) 2025-10-10T01:00:00.5261476Z 2025-10-10T01:00:00.5261574Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5261664Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5261783Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5261995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5262089Z res = mod(**inputs) 2025-10-10T01:00:00.5262422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5262499Z outputs = self.mobilebert( 2025-10-10T01:00:00.5262805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5262883Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5263181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5263267Z layer_outputs = layer_module( 2025-10-10T01:00:00.5263563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5263661Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5263982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5264147Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5264443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.5264537Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5264541Z 2025-10-10T01:00:00.5264661Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5264878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5264957Z res = mod(**inputs) 2025-10-10T01:00:00.5265261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5265343Z outputs = self.mobilebert( 2025-10-10T01:00:00.5265664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5265745Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5266060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5266137Z layer_outputs = layer_module( 2025-10-10T01:00:00.5266523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5266629Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5266947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5267100Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5267412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.5267567Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5267875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5267982Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5268001Z 2025-10-10T01:00:00.5268111Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5268311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5268385Z res = mod(**inputs) 2025-10-10T01:00:00.5268666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5268742Z outputs = self.mobilebert( 2025-10-10T01:00:00.5269030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5269145Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5269432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5269504Z layer_outputs = layer_module( 2025-10-10T01:00:00.5269794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5269892Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5270175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5270298Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5270581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5270706Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5270712Z 2025-10-10T01:00:00.5270818Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5271024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5271088Z res = mod(**inputs) 2025-10-10T01:00:00.5271368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5271446Z outputs = self.mobilebert( 2025-10-10T01:00:00.5271726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5271805Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5272084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5272162Z layer_outputs = layer_module( 2025-10-10T01:00:00.5272453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5272550Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5272841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5272954Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5273241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5273357Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5273361Z 2025-10-10T01:00:00.5273466Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5273672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5273743Z res = mod(**inputs) 2025-10-10T01:00:00.5274034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5274106Z outputs = self.mobilebert( 2025-10-10T01:00:00.5274395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5274469Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5274768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5274854Z layer_outputs = layer_module( 2025-10-10T01:00:00.5275154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5275271Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5275618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5275750Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5276038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5276131Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5276134Z 2025-10-10T01:00:00.5276246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5276445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5276523Z res = mod(**inputs) 2025-10-10T01:00:00.5276826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5276903Z outputs = self.mobilebert( 2025-10-10T01:00:00.5277256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5277337Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5277647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5277722Z layer_outputs = layer_module( 2025-10-10T01:00:00.5278027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5278138Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5278454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5278587Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5278876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5279006Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5279287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5279381Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5279384Z 2025-10-10T01:00:00.5279495Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5279696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5279768Z res = mod(**inputs) 2025-10-10T01:00:00.5280044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5280114Z outputs = self.mobilebert( 2025-10-10T01:00:00.5280404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5280481Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5280766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5280835Z layer_outputs = layer_module( 2025-10-10T01:00:00.5281119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5281213Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5281497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5281622Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5281916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5282051Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5282054Z 2025-10-10T01:00:00.5282161Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5282361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5282434Z res = mod(**inputs) 2025-10-10T01:00:00.5282716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5282798Z outputs = self.mobilebert( 2025-10-10T01:00:00.5283078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5283161Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5283447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5283565Z layer_outputs = layer_module( 2025-10-10T01:00:00.5283849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5283941Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5284220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5284329Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5284612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5284730Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5284734Z 2025-10-10T01:00:00.5284836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5285041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5285108Z res = mod(**inputs) 2025-10-10T01:00:00.5285469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5285553Z outputs = self.mobilebert( 2025-10-10T01:00:00.5285832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5285914Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5286200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5286279Z layer_outputs = layer_module( 2025-10-10T01:00:00.5286566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5286660Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5286951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5287079Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5287368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5287455Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5287461Z 2025-10-10T01:00:00.5287572Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5287771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5287838Z res = mod(**inputs) 2025-10-10T01:00:00.5288128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5288242Z outputs = self.mobilebert( 2025-10-10T01:00:00.5288570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5288650Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5288947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5289030Z layer_outputs = layer_module( 2025-10-10T01:00:00.5289328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5289438Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5289734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5289868Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5290197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5290322Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5290611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5290705Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5290708Z 2025-10-10T01:00:00.5290818Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5291015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5291089Z res = mod(**inputs) 2025-10-10T01:00:00.5291373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5291449Z outputs = self.mobilebert( 2025-10-10T01:00:00.5291746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5291820Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5292109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5292182Z layer_outputs = layer_module( 2025-10-10T01:00:00.5292462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5292565Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5292848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5292971Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5293256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5293355Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5293359Z 2025-10-10T01:00:00.5293463Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5293661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5293736Z res = mod(**inputs) 2025-10-10T01:00:00.5294015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5294093Z outputs = self.mobilebert( 2025-10-10T01:00:00.5294373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5294446Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5294738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5294851Z layer_outputs = layer_module( 2025-10-10T01:00:00.5295151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5295252Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5295553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5295674Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5295981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5296108Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5296112Z 2025-10-10T01:00:00.5296224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5296464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5296552Z res = mod(**inputs) 2025-10-10T01:00:00.5296853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5296939Z outputs = self.mobilebert( 2025-10-10T01:00:00.5297239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5297321Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5297620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5297702Z layer_outputs = layer_module( 2025-10-10T01:00:00.5298003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5298123Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5298417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5298543Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5298837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5298925Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5298929Z 2025-10-10T01:00:00.5299033Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5299240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5299307Z res = mod(**inputs) 2025-10-10T01:00:00.5299600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5299677Z outputs = self.mobilebert( 2025-10-10T01:00:00.5299980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5300060Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5300359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5300444Z layer_outputs = layer_module( 2025-10-10T01:00:00.5300743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5300853Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5301151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5301285Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5301631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5301765Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5302069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5302168Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5302172Z 2025-10-10T01:00:00.5302289Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5302510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5302581Z res = mod(**inputs) 2025-10-10T01:00:00.5302887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5302967Z outputs = self.mobilebert( 2025-10-10T01:00:00.5303321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5303401Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5303706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5303790Z layer_outputs = layer_module( 2025-10-10T01:00:00.5304094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5304234Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5304545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5304646Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5304653Z 2025-10-10T01:00:00.5304770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5304992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5305071Z res = mod(**inputs) 2025-10-10T01:00:00.5305381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5305470Z outputs = self.mobilebert( 2025-10-10T01:00:00.5305776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5305857Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5306169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5306248Z layer_outputs = layer_module( 2025-10-10T01:00:00.5306637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5306781Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5307095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5307218Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5307222Z 2025-10-10T01:00:00.5307336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5307579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5307649Z res = mod(**inputs) 2025-10-10T01:00:00.5307956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5308035Z outputs = self.mobilebert( 2025-10-10T01:00:00.5308343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5308482Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5308792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5308878Z layer_outputs = layer_module( 2025-10-10T01:00:00.5309187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5309370Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5309680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.5309786Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.5309792Z 2025-10-10T01:00:00.5309914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5310168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5310249Z res = mod(**inputs) 2025-10-10T01:00:00.5310556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5310640Z outputs = self.mobilebert( 2025-10-10T01:00:00.5310945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5311025Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5311340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5311418Z layer_outputs = layer_module( 2025-10-10T01:00:00.5311733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5311912Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5312221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.5312365Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.5312676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5312785Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5312789Z 2025-10-10T01:00:00.5312903Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5313130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5313202Z res = mod(**inputs) 2025-10-10T01:00:00.5313509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5313599Z outputs = self.mobilebert( 2025-10-10T01:00:00.5313911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5314001Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5314306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5314384Z layer_outputs = layer_module( 2025-10-10T01:00:00.5314702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5314876Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5315193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5315358Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5315694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.5315792Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5315796Z 2025-10-10T01:00:00.5315909Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5316140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5316211Z res = mod(**inputs) 2025-10-10T01:00:00.5316532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5316610Z outputs = self.mobilebert( 2025-10-10T01:00:00.5316921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5317052Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5317357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5317445Z layer_outputs = layer_module( 2025-10-10T01:00:00.5317748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5317927Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5318233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5318370Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5318683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5318822Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5319140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5319244Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5319248Z 2025-10-10T01:00:00.5319369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5319597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5319669Z res = mod(**inputs) 2025-10-10T01:00:00.5319982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5320060Z outputs = self.mobilebert( 2025-10-10T01:00:00.5320374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5320459Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5320767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5320853Z layer_outputs = layer_module( 2025-10-10T01:00:00.5321161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5321345Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5321657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5321784Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5322096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5322217Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5322221Z 2025-10-10T01:00:00.5322360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5322576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5322652Z res = mod(**inputs) 2025-10-10T01:00:00.5322961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5323038Z outputs = self.mobilebert( 2025-10-10T01:00:00.5323352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5323433Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5323747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5323824Z layer_outputs = layer_module( 2025-10-10T01:00:00.5324177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5324277Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5324584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5324671Z self_outputs = self.self( 2025-10-10T01:00:00.5324979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-10-10T01:00:00.5325065Z self.value(value_tensor) 2025-10-10T01:00:00.5325069Z 2025-10-10T01:00:00.5325184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5325411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5325490Z res = mod(**inputs) 2025-10-10T01:00:00.5325807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5325892Z outputs = self.mobilebert( 2025-10-10T01:00:00.5326201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5326289Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5326597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5326676Z layer_outputs = layer_module( 2025-10-10T01:00:00.5326990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5327168Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5327487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-10-10T01:00:00.5327614Z bottlenecked_hidden_states = self.input(hidden_states) 2025-10-10T01:00:00.5327930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-10-10T01:00:00.5328032Z layer_input = self.dense(hidden_states) 2025-10-10T01:00:00.5328036Z 2025-10-10T01:00:00.5328160Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5328386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5328457Z res = mod(**inputs) 2025-10-10T01:00:00.5328761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5328836Z outputs = self.mobilebert( 2025-10-10T01:00:00.5329138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5329263Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5329563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5329645Z layer_outputs = layer_module( 2025-10-10T01:00:00.5329943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-10-10T01:00:00.5330112Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-10-10T01:00:00.5330417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-10-10T01:00:00.5330536Z shared_attention_input = self.attention(hidden_states) 2025-10-10T01:00:00.5330865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-10-10T01:00:00.5330980Z layer_input = self.LayerNorm(layer_input) 2025-10-10T01:00:00.5331283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5331382Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5331385Z 2025-10-10T01:00:00.5331686Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5331922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5331993Z res = mod(**inputs) 2025-10-10T01:00:00.5332301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5332377Z outputs = self.mobilebert( 2025-10-10T01:00:00.5332679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5332774Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5333076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5333161Z layer_outputs = layer_module( 2025-10-10T01:00:00.5333461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5333565Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5333866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5333944Z self_outputs = self.self( 2025-10-10T01:00:00.5334252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-10-10T01:00:00.5334332Z self.query(query_tensor) 2025-10-10T01:00:00.5334339Z 2025-10-10T01:00:00.5334460Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5334669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5334743Z res = mod(**inputs) 2025-10-10T01:00:00.5335054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5335129Z outputs = self.mobilebert( 2025-10-10T01:00:00.5335436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5335513Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5335820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5335897Z layer_outputs = layer_module( 2025-10-10T01:00:00.5336292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5336401Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5336710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-10-10T01:00:00.5336807Z self_outputs = self.self( 2025-10-10T01:00:00.5337110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-10-10T01:00:00.5337183Z self.key(key_tensor) 2025-10-10T01:00:00.5337194Z 2025-10-10T01:00:00.5337284Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5337373Z cudagraph partition due to non gpu ops 2025-10-10T01:00:00.5337492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5337705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5337826Z res = mod(**inputs) 2025-10-10T01:00:00.5338136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5338214Z outputs = self.mobilebert( 2025-10-10T01:00:00.5338526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5338605Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5338921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5338999Z layer_outputs = layer_module( 2025-10-10T01:00:00.5339307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5339409Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5339722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5339871Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5340183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-10-10T01:00:00.5340276Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5340288Z 2025-10-10T01:00:00.5340401Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5340620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5340699Z res = mod(**inputs) 2025-10-10T01:00:00.5341008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5341092Z outputs = self.mobilebert( 2025-10-10T01:00:00.5341405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5341485Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5341802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5341882Z layer_outputs = layer_module( 2025-10-10T01:00:00.5342198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-10-10T01:00:00.5342293Z self_attention_outputs = self.attention( 2025-10-10T01:00:00.5342599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-10-10T01:00:00.5342742Z attention_output = self.output(self_outputs[0], layer_input) 2025-10-10T01:00:00.5343054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-10-10T01:00:00.5343250Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5343559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5343667Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5343671Z 2025-10-10T01:00:00.5343787Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5344004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5344087Z res = mod(**inputs) 2025-10-10T01:00:00.5344394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5344479Z outputs = self.mobilebert( 2025-10-10T01:00:00.5344807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5344912Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5345219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5345297Z layer_outputs = layer_module( 2025-10-10T01:00:00.5345611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5345720Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5346032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5346161Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5346587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5346717Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5346723Z 2025-10-10T01:00:00.5346839Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5347067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5347141Z res = mod(**inputs) 2025-10-10T01:00:00.5347463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5347544Z outputs = self.mobilebert( 2025-10-10T01:00:00.5347859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5347948Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5348263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5348354Z layer_outputs = layer_module( 2025-10-10T01:00:00.5348670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5348777Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5349100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5349224Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5349543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5349670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5349674Z 2025-10-10T01:00:00.5349795Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5350014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5350124Z res = mod(**inputs) 2025-10-10T01:00:00.5350463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5350542Z outputs = self.mobilebert( 2025-10-10T01:00:00.5350856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5350938Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5351243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5351329Z layer_outputs = layer_module( 2025-10-10T01:00:00.5351634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5351746Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5352092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5352240Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5352548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5352642Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5352646Z 2025-10-10T01:00:00.5352769Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5352986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5353067Z res = mod(**inputs) 2025-10-10T01:00:00.5353376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5353453Z outputs = self.mobilebert( 2025-10-10T01:00:00.5353784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5353863Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5354182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5354262Z layer_outputs = layer_module( 2025-10-10T01:00:00.5354580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5354681Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5354977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5355121Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5355423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5355568Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5355868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5355968Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5355979Z 2025-10-10T01:00:00.5356088Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5356300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5356378Z res = mod(**inputs) 2025-10-10T01:00:00.5356677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5356760Z outputs = self.mobilebert( 2025-10-10T01:00:00.5357055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5357173Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5357478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5357553Z layer_outputs = layer_module( 2025-10-10T01:00:00.5357858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5357958Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5358259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5358388Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5358704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5358834Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5358838Z 2025-10-10T01:00:00.5358948Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5359166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5359237Z res = mod(**inputs) 2025-10-10T01:00:00.5359541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5359624Z outputs = self.mobilebert( 2025-10-10T01:00:00.5359929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5360013Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5360307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5360388Z layer_outputs = layer_module( 2025-10-10T01:00:00.5360693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5360793Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5361100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5361219Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5361525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5361645Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5361649Z 2025-10-10T01:00:00.5361759Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5361990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5362064Z res = mod(**inputs) 2025-10-10T01:00:00.5362373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5362447Z outputs = self.mobilebert( 2025-10-10T01:00:00.5362750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5362828Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5363127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5363215Z layer_outputs = layer_module( 2025-10-10T01:00:00.5363512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5363623Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5363962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5364099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5364409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5364500Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5364504Z 2025-10-10T01:00:00.5364620Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5364841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5364916Z res = mod(**inputs) 2025-10-10T01:00:00.5365213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5365290Z outputs = self.mobilebert( 2025-10-10T01:00:00.5365630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5365709Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5366019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5366094Z layer_outputs = layer_module( 2025-10-10T01:00:00.5366406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5366514Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5366812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5366950Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5367251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5367390Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5367713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5367812Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5367816Z 2025-10-10T01:00:00.5367934Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5368155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5368232Z res = mod(**inputs) 2025-10-10T01:00:00.5368526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5368603Z outputs = self.mobilebert( 2025-10-10T01:00:00.5368913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5368993Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5369297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5369376Z layer_outputs = layer_module( 2025-10-10T01:00:00.5369679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5369779Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5370076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5370204Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5370502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5370640Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5370644Z 2025-10-10T01:00:00.5370756Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5370964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5371040Z res = mod(**inputs) 2025-10-10T01:00:00.5371336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5371418Z outputs = self.mobilebert( 2025-10-10T01:00:00.5371718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5371803Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5372115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5372210Z layer_outputs = layer_module( 2025-10-10T01:00:00.5372521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5372620Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5372924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-10-10T01:00:00.5373051Z intermediate_output = self.intermediate(hidden_states) 2025-10-10T01:00:00.5373354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5373482Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5373486Z 2025-10-10T01:00:00.5373594Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5373812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5373884Z res = mod(**inputs) 2025-10-10T01:00:00.5374188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5374264Z outputs = self.mobilebert( 2025-10-10T01:00:00.5374561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5374645Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5374940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5375021Z layer_outputs = layer_module( 2025-10-10T01:00:00.5375322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5375427Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5375735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5375868Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5376173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-10-10T01:00:00.5376264Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5376267Z 2025-10-10T01:00:00.5376385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5376596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5376666Z res = mod(**inputs) 2025-10-10T01:00:00.5376972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5377069Z outputs = self.mobilebert( 2025-10-10T01:00:00.5377392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5377474Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5377771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5377855Z layer_outputs = layer_module( 2025-10-10T01:00:00.5378152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-10-10T01:00:00.5378262Z attention_output = ffn_module(attention_output) 2025-10-10T01:00:00.5378561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-10-10T01:00:00.5378701Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-10-10T01:00:00.5379063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-10-10T01:00:00.5379198Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5379510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5379615Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5379619Z 2025-10-10T01:00:00.5379740Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5379955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5380036Z res = mod(**inputs) 2025-10-10T01:00:00.5380341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5380422Z outputs = self.mobilebert( 2025-10-10T01:00:00.5380745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5380826Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5381138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5381218Z layer_outputs = layer_module( 2025-10-10T01:00:00.5381526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5381669Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5381970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-10-10T01:00:00.5382071Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5382074Z 2025-10-10T01:00:00.5382193Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5382421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5382494Z res = mod(**inputs) 2025-10-10T01:00:00.5382796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5382883Z outputs = self.mobilebert( 2025-10-10T01:00:00.5383199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5383290Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5383601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5383685Z layer_outputs = layer_module( 2025-10-10T01:00:00.5384008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-10-10T01:00:00.5384182Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:00:00.5384492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-10-10T01:00:00.5384614Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:00:00.5384618Z 2025-10-10T01:00:00.5384737Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5384953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5385025Z res = mod(**inputs) 2025-10-10T01:00:00.5385342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5385422Z outputs = self.mobilebert( 2025-10-10T01:00:00.5385763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5385869Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5386179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5386265Z layer_outputs = layer_module( 2025-10-10T01:00:00.5386643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5386833Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5387145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-10-10T01:00:00.5387260Z layer_output = self.dense(intermediate_states) 2025-10-10T01:00:00.5387264Z 2025-10-10T01:00:00.5387377Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5387607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5387689Z res = mod(**inputs) 2025-10-10T01:00:00.5387997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5388083Z outputs = self.mobilebert( 2025-10-10T01:00:00.5388389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5388469Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5388783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5388862Z layer_outputs = layer_module( 2025-10-10T01:00:00.5389175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5389356Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5389674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-10-10T01:00:00.5389812Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-10-10T01:00:00.5390116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5390230Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5390234Z 2025-10-10T01:00:00.5390346Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5390574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5390648Z res = mod(**inputs) 2025-10-10T01:00:00.5390953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5391083Z outputs = self.mobilebert( 2025-10-10T01:00:00.5391395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5391482Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5391796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5391880Z layer_outputs = layer_module( 2025-10-10T01:00:00.5392194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5392367Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5392690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5392866Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5393181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-10-10T01:00:00.5393275Z layer_outputs = self.dense(hidden_states) 2025-10-10T01:00:00.5393279Z 2025-10-10T01:00:00.5393400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5393617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5393688Z res = mod(**inputs) 2025-10-10T01:00:00.5394002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-10-10T01:00:00.5394081Z outputs = self.mobilebert( 2025-10-10T01:00:00.5394393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-10-10T01:00:00.5394477Z encoder_outputs = self.encoder( 2025-10-10T01:00:00.5394785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-10-10T01:00:00.5394871Z layer_outputs = layer_module( 2025-10-10T01:00:00.5395177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-10-10T01:00:00.5395355Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-10-10T01:00:00.5395661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-10-10T01:00:00.5395806Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-10-10T01:00:00.5396112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-10-10T01:00:00.5396253Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-10-10T01:00:00.5396570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-10-10T01:00:00.5396674Z return input_tensor * self.weight + self.bias 2025-10-10T01:00:00.5396678Z 2025-10-10T01:00:00.5396799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5397016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5397090Z res = mod(**inputs) 2025-10-10T01:00:00.5397407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-10-10T01:00:00.5397515Z prediction_scores = self.cls(sequence_output) 2025-10-10T01:00:00.5397828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-10-10T01:00:00.5397991Z prediction_scores = self.predictions(sequence_output) 2025-10-10T01:00:00.5398318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-10-10T01:00:00.5398420Z hidden_states = self.transform(hidden_states) 2025-10-10T01:00:00.5398718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-10-10T01:00:00.5398818Z hidden_states = self.dense(hidden_states) 2025-10-10T01:00:00.5398821Z 2025-10-10T01:00:00.5398932Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5399151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5399220Z res = mod(**inputs) 2025-10-10T01:00:00.5399523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-10-10T01:00:00.5399662Z prediction_scores = self.cls(sequence_output) 2025-10-10T01:00:00.5399962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-10-10T01:00:00.5400090Z prediction_scores = self.predictions(sequence_output) 2025-10-10T01:00:00.5400388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-10-10T01:00:00.5400618Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-10-10T01:00:00.5400623Z 2025-10-10T01:00:00.5400733Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5400942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5401023Z res = mod(**inputs) 2025-10-10T01:00:00.5401324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-10-10T01:00:00.5401436Z prediction_scores = self.cls(sequence_output) 2025-10-10T01:00:00.5401733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-10-10T01:00:00.5401859Z prediction_scores = self.predictions(sequence_output) 2025-10-10T01:00:00.5402156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-10-10T01:00:00.5402245Z hidden_states += self.decoder.bias 2025-10-10T01:00:00.5402249Z 2025-10-10T01:00:00.5402367Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:00.5402577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:00.5402655Z res = mod(**inputs) 2025-10-10T01:00:00.5402966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-10-10T01:00:00.5403183Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T01:00:00.5403195Z 2025-10-10T01:00:15.4199965Z Compilation time (from dynamo_timed): 41.366440906 2025-10-10T01:00:15.4200288Z pass 2025-10-10T01:00:15.4200619Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:15.4201471Z TIMING: _recursive_pre_grad_passes:0.023 _recursive_joint_graph_passes:1.98126 _recursive_post_grad_passes:0.21168 async_compile.wait:0.75061 code_gen:11.68838 inductor_compile:16.25016 backend_compile:28.65045 gc:0.00064 entire_frame_compile:41.36644 total_wall_time:41.36644 2025-10-10T01:00:15.4202477Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:31168 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:8820 2025-10-10T01:00:15.4203333Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-10-10T01:00:19.4873173Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:00:19.4874088Z import pynvml # type: ignore[import] 2025-10-10T01:00:23.0380926Z 2025-10-10T01:00:24.8902995Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:00:24.8903330Z loading model: 0it [00:01, ?it/s] 2025-10-10T01:00:24.8912494Z cpu eval OPTForCausalLM 2025-10-10T01:00:27.0360587Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:28.0049646Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:28.9820327Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:36.7823037Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7823381Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7823632Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7823875Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7824118Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7824349Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7824589Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7824831Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7825060Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7825285Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7825512Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7825743Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7826014Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7826532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7826950Z res = mod(**inputs) 2025-10-10T01:00:36.7827380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7827919Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7828376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7828858Z outputs = self.model.decoder( 2025-10-10T01:00:36.7829257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7829688Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7830124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7830518Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7830932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7831359Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7831985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7832435Z return func(*args, **kwargs) 2025-10-10T01:00:36.7832858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7833339Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7833764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7834196Z return func(*args, **kwargs) 2025-10-10T01:00:36.7834662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.7835464Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.7835667Z 2025-10-10T01:00:36.7835853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7836291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7836673Z res = mod(**inputs) 2025-10-10T01:00:36.7837056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7837463Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7837906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7838345Z outputs = self.model.decoder( 2025-10-10T01:00:36.7838741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7839136Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7839618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7840104Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7840525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7840951Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7841392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7841827Z return func(*args, **kwargs) 2025-10-10T01:00:36.7842254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7842730Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7843198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7843630Z return func(*args, **kwargs) 2025-10-10T01:00:36.7844055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.7844524Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.7844674Z 2025-10-10T01:00:36.7844931Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7845331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7845696Z res = mod(**inputs) 2025-10-10T01:00:36.7846075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7846477Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7846905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7847345Z outputs = self.model.decoder( 2025-10-10T01:00:36.7847744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7848121Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7848516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7848913Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7849278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7849680Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7850081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7850501Z return func(*args, **kwargs) 2025-10-10T01:00:36.7850919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7851379Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7851875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7852280Z return func(*args, **kwargs) 2025-10-10T01:00:36.7852650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.7853041Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.7853196Z 2025-10-10T01:00:36.7853281Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7853501Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7853746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7854115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7854440Z res = mod(**inputs) 2025-10-10T01:00:36.7854779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7855183Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7855572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7855954Z outputs = self.model.decoder( 2025-10-10T01:00:36.7856317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7856683Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7857072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7857459Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7857827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7858209Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7858605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7859002Z return func(*args, **kwargs) 2025-10-10T01:00:36.7859383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7859792Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7860214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7860631Z return func(*args, **kwargs) 2025-10-10T01:00:36.7861033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.7861528Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.7862029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.7862574Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.7862785Z 2025-10-10T01:00:36.7862916Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7863320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7863670Z res = mod(**inputs) 2025-10-10T01:00:36.7864037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7864452Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7864887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7865318Z outputs = self.model.decoder( 2025-10-10T01:00:36.7865709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7866112Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7866776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7867204Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7867598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7868005Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7868420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7868827Z return func(*args, **kwargs) 2025-10-10T01:00:36.7869225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7869654Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7870089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7870537Z return func(*args, **kwargs) 2025-10-10T01:00:36.7870934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.7871361Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.7871519Z 2025-10-10T01:00:36.7871649Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7872040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7872387Z res = mod(**inputs) 2025-10-10T01:00:36.7872740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7873116Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7873524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7873934Z outputs = self.model.decoder( 2025-10-10T01:00:36.7874334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7874723Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7875118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7875527Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7875961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7876366Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7876779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7877178Z return func(*args, **kwargs) 2025-10-10T01:00:36.7877574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.7878005Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.7878164Z 2025-10-10T01:00:36.7878288Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7878683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7879035Z res = mod(**inputs) 2025-10-10T01:00:36.7879389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7879782Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7880193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7880594Z outputs = self.model.decoder( 2025-10-10T01:00:36.7880967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7881359Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7881820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7882242Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7882626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7883021Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7883436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7883847Z return func(*args, **kwargs) 2025-10-10T01:00:36.7884233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.7884677Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.7884850Z 2025-10-10T01:00:36.7884962Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7885390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7885784Z res = mod(**inputs) 2025-10-10T01:00:36.7886141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7886529Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7886950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7887367Z outputs = self.model.decoder( 2025-10-10T01:00:36.7887742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7888135Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7888541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7888961Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7889394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7889791Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7890215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7890624Z return func(*args, **kwargs) 2025-10-10T01:00:36.7891022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.7891444Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.7891602Z 2025-10-10T01:00:36.7891716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7892117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7892473Z res = mod(**inputs) 2025-10-10T01:00:36.7892833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7893225Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7893650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7894065Z outputs = self.model.decoder( 2025-10-10T01:00:36.7894455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7894844Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7895248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7895651Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7896029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7896420Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7896824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7897285Z return func(*args, **kwargs) 2025-10-10T01:00:36.7897681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7898128Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7898564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7898972Z return func(*args, **kwargs) 2025-10-10T01:00:36.7899366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.7899822Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.7899999Z 2025-10-10T01:00:36.7900120Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7900526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7900889Z res = mod(**inputs) 2025-10-10T01:00:36.7901246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7901628Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7902032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7902430Z outputs = self.model.decoder( 2025-10-10T01:00:36.7902798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7903180Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7903584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7903996Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7904370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7904767Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7905184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7905591Z return func(*args, **kwargs) 2025-10-10T01:00:36.7905975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7906496Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7906936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7907338Z return func(*args, **kwargs) 2025-10-10T01:00:36.7907730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.7908144Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.7908307Z 2025-10-10T01:00:36.7908424Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7908814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7909164Z res = mod(**inputs) 2025-10-10T01:00:36.7909524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7909905Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7910315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7910721Z outputs = self.model.decoder( 2025-10-10T01:00:36.7911097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7911472Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7911879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7912336Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7912732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7913142Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7913547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7913953Z return func(*args, **kwargs) 2025-10-10T01:00:36.7914345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7914775Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7915204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7915600Z return func(*args, **kwargs) 2025-10-10T01:00:36.7916028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.7916448Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.7916601Z 2025-10-10T01:00:36.7916700Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7916928Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7917187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7917578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7917929Z res = mod(**inputs) 2025-10-10T01:00:36.7918281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7918676Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7919081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7919492Z outputs = self.model.decoder( 2025-10-10T01:00:36.7919867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7920241Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7920739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7921151Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7921533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7921925Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7922329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7922731Z return func(*args, **kwargs) 2025-10-10T01:00:36.7923128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7923567Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7923990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7924390Z return func(*args, **kwargs) 2025-10-10T01:00:36.7924784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.7925221Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.7925706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.7926226Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.7926434Z 2025-10-10T01:00:36.7926549Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7926972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7927371Z res = mod(**inputs) 2025-10-10T01:00:36.7927731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7928112Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7928562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7928973Z outputs = self.model.decoder( 2025-10-10T01:00:36.7929349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7929725Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7930128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7930567Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7930974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7931387Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7932000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7932445Z return func(*args, **kwargs) 2025-10-10T01:00:36.7932819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7933231Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7933645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7934022Z return func(*args, **kwargs) 2025-10-10T01:00:36.7934415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.7934853Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.7935006Z 2025-10-10T01:00:36.7935133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7935517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7935876Z res = mod(**inputs) 2025-10-10T01:00:36.7936232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7936600Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7936983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7937362Z outputs = self.model.decoder( 2025-10-10T01:00:36.7937719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7938079Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7938470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7938847Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7939208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7939579Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7939971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7940354Z return func(*args, **kwargs) 2025-10-10T01:00:36.7940719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.7941112Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.7941260Z 2025-10-10T01:00:36.7941369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7941735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7942170Z res = mod(**inputs) 2025-10-10T01:00:36.7942499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7942864Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7943247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7943632Z outputs = self.model.decoder( 2025-10-10T01:00:36.7943978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7944345Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7944727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7945114Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7945518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7945938Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7946398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7946815Z return func(*args, **kwargs) 2025-10-10T01:00:36.7947210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.7947647Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.7947821Z 2025-10-10T01:00:36.7947929Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7948293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7948620Z res = mod(**inputs) 2025-10-10T01:00:36.7948955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7949320Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7949712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7950106Z outputs = self.model.decoder( 2025-10-10T01:00:36.7950484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7950864Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7951264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7951671Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7952060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7952431Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7952831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7953287Z return func(*args, **kwargs) 2025-10-10T01:00:36.7953658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.7954054Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.7954197Z 2025-10-10T01:00:36.7954310Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7954669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7954997Z res = mod(**inputs) 2025-10-10T01:00:36.7955352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7955771Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7956155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7956553Z outputs = self.model.decoder( 2025-10-10T01:00:36.7956923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7957282Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7957667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7958044Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7958402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7958775Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7959185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7959589Z return func(*args, **kwargs) 2025-10-10T01:00:36.7959990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7960440Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7960853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7961236Z return func(*args, **kwargs) 2025-10-10T01:00:36.7961605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.7962037Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.7962214Z 2025-10-10T01:00:36.7962323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7962697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7963030Z res = mod(**inputs) 2025-10-10T01:00:36.7963367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7963732Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7964115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7964500Z outputs = self.model.decoder( 2025-10-10T01:00:36.7964857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7965234Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7965652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7966059Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7966435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7966800Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7967196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7967601Z return func(*args, **kwargs) 2025-10-10T01:00:36.7967999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7968433Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7968858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7969263Z return func(*args, **kwargs) 2025-10-10T01:00:36.7969655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.7970072Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.7970224Z 2025-10-10T01:00:36.7970345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7970729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7971108Z res = mod(**inputs) 2025-10-10T01:00:36.7971481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7971868Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7972264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7972673Z outputs = self.model.decoder( 2025-10-10T01:00:36.7973044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7973424Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7973826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7974224Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7974625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7975038Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7975451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7975844Z return func(*args, **kwargs) 2025-10-10T01:00:36.7976233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7976660Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7977091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7977491Z return func(*args, **kwargs) 2025-10-10T01:00:36.7977877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.7978295Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.7978458Z 2025-10-10T01:00:36.7978548Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7978793Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.7979047Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7979435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7979787Z res = mod(**inputs) 2025-10-10T01:00:36.7980143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7980527Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7980930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7981336Z outputs = self.model.decoder( 2025-10-10T01:00:36.7981708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7982093Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7982497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7982893Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7983277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7983679Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7984100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7984494Z return func(*args, **kwargs) 2025-10-10T01:00:36.7984885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7985329Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7985773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7987334Z return func(*args, **kwargs) 2025-10-10T01:00:36.7987744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.7988158Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.7988625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.7989125Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.7989315Z 2025-10-10T01:00:36.7989431Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7989796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7990125Z res = mod(**inputs) 2025-10-10T01:00:36.7990483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7990872Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7991251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7991637Z outputs = self.model.decoder( 2025-10-10T01:00:36.7992008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7992400Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7992782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.7993159Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.7993517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.7993888Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.7994285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7994664Z return func(*args, **kwargs) 2025-10-10T01:00:36.7995034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.7995446Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.7995855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.7996247Z return func(*args, **kwargs) 2025-10-10T01:00:36.7996629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.7997044Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.7997200Z 2025-10-10T01:00:36.7997320Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.7997689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.7998023Z res = mod(**inputs) 2025-10-10T01:00:36.7998353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.7998718Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.7999102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.7999489Z outputs = self.model.decoder( 2025-10-10T01:00:36.7999837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8000200Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8000603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8001009Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8001428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8001838Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8002250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8002632Z return func(*args, **kwargs) 2025-10-10T01:00:36.8003004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8003454Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8003603Z 2025-10-10T01:00:36.8003717Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8004104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8004446Z res = mod(**inputs) 2025-10-10T01:00:36.8004800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8005211Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8005621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8006027Z outputs = self.model.decoder( 2025-10-10T01:00:36.8006399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8006782Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8007178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8007583Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8007961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8008351Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8008753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8009156Z return func(*args, **kwargs) 2025-10-10T01:00:36.8009549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8009980Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8010146Z 2025-10-10T01:00:36.8010264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8010641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8010986Z res = mod(**inputs) 2025-10-10T01:00:36.8011336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8011717Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8012118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8012520Z outputs = self.model.decoder( 2025-10-10T01:00:36.8012894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8013276Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8013680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8014076Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8014457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8014848Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8015260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8015663Z return func(*args, **kwargs) 2025-10-10T01:00:36.8016052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8016511Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8016670Z 2025-10-10T01:00:36.8016783Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8017171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8017516Z res = mod(**inputs) 2025-10-10T01:00:36.8017871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8018251Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8018655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8019057Z outputs = self.model.decoder( 2025-10-10T01:00:36.8019421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8019869Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8020274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8020678Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8021055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8021440Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8021851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8022254Z return func(*args, **kwargs) 2025-10-10T01:00:36.8022644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-10-10T01:00:36.8023106Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-10-10T01:00:36.8023321Z 2025-10-10T01:00:36.8023438Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8023831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8024181Z res = mod(**inputs) 2025-10-10T01:00:36.8024517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8024891Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8025300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8025708Z outputs = self.model.decoder( 2025-10-10T01:00:36.8026081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8026552Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8026964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8027388Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8027780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8028184Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8028600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8029019Z return func(*args, **kwargs) 2025-10-10T01:00:36.8029419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8029868Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8030310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8030714Z return func(*args, **kwargs) 2025-10-10T01:00:36.8031118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8031784Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8031969Z 2025-10-10T01:00:36.8032092Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8032487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8032833Z res = mod(**inputs) 2025-10-10T01:00:36.8033192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8033576Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8033985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8034373Z outputs = self.model.decoder( 2025-10-10T01:00:36.8034740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8035189Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8035608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8036031Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8036401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8036799Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8037265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8037660Z return func(*args, **kwargs) 2025-10-10T01:00:36.8038035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8038469Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8038888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8039282Z return func(*args, **kwargs) 2025-10-10T01:00:36.8039657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8040052Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8040208Z 2025-10-10T01:00:36.8040311Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8040668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8040987Z res = mod(**inputs) 2025-10-10T01:00:36.8041313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8041661Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8042034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8042412Z outputs = self.model.decoder( 2025-10-10T01:00:36.8042759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8043104Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8043489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8043868Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8044226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8044598Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8044969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8045341Z return func(*args, **kwargs) 2025-10-10T01:00:36.8045696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8046131Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8046526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8046903Z return func(*args, **kwargs) 2025-10-10T01:00:36.8047262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8047650Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8047786Z 2025-10-10T01:00:36.8047874Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8048078Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8048310Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8048660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8048973Z res = mod(**inputs) 2025-10-10T01:00:36.8049330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8049677Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8050044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8050408Z outputs = self.model.decoder( 2025-10-10T01:00:36.8050743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8051080Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8051444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8051818Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8052169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8052541Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8052905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8053270Z return func(*args, **kwargs) 2025-10-10T01:00:36.8053626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8054018Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8054404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8054777Z return func(*args, **kwargs) 2025-10-10T01:00:36.8055134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8055544Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8056009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8056494Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8056685Z 2025-10-10T01:00:36.8056789Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8057150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8057475Z res = mod(**inputs) 2025-10-10T01:00:36.8057809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8058167Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8058554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8058936Z outputs = self.model.decoder( 2025-10-10T01:00:36.8059294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8059674Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8060076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8060460Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8060819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8061188Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8061569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8061957Z return func(*args, **kwargs) 2025-10-10T01:00:36.8062345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8062775Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8063222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8063639Z return func(*args, **kwargs) 2025-10-10T01:00:36.8064027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8064439Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8064588Z 2025-10-10T01:00:36.8064709Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8065091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8065444Z res = mod(**inputs) 2025-10-10T01:00:36.8065803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8066200Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8066705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8067124Z outputs = self.model.decoder( 2025-10-10T01:00:36.8067511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8067915Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8068301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8068680Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8069042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8069412Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8069802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8070183Z return func(*args, **kwargs) 2025-10-10T01:00:36.8070547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8070943Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8071092Z 2025-10-10T01:00:36.8071197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8071560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8071888Z res = mod(**inputs) 2025-10-10T01:00:36.8072215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8072574Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8072958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8073335Z outputs = self.model.decoder( 2025-10-10T01:00:36.8073678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8074085Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8074484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8074859Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8075209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8075569Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8075955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8076338Z return func(*args, **kwargs) 2025-10-10T01:00:36.8076709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8077113Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8077282Z 2025-10-10T01:00:36.8077431Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8077797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8078110Z res = mod(**inputs) 2025-10-10T01:00:36.8078427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8078761Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8079124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8079487Z outputs = self.model.decoder( 2025-10-10T01:00:36.8079835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8080195Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8080569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8080954Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8081315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8081682Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8082045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8082405Z return func(*args, **kwargs) 2025-10-10T01:00:36.8082754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8083130Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8083270Z 2025-10-10T01:00:36.8083381Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8083742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8084075Z res = mod(**inputs) 2025-10-10T01:00:36.8084408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8084761Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8085128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8085504Z outputs = self.model.decoder( 2025-10-10T01:00:36.8085838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8086180Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8086548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8086911Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8087259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8087649Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8088045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8088421Z return func(*args, **kwargs) 2025-10-10T01:00:36.8088785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8089194Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8089587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8089957Z return func(*args, **kwargs) 2025-10-10T01:00:36.8090314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8090742Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8090925Z 2025-10-10T01:00:36.8091060Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8091475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8091828Z res = mod(**inputs) 2025-10-10T01:00:36.8092170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8092537Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8092926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8093317Z outputs = self.model.decoder( 2025-10-10T01:00:36.8093671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8094035Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8094440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8094848Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8095214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8095581Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8095980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8096382Z return func(*args, **kwargs) 2025-10-10T01:00:36.8096770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8097181Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8097585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8097967Z return func(*args, **kwargs) 2025-10-10T01:00:36.8098342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8098742Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8098882Z 2025-10-10T01:00:36.8098997Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8099359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8099696Z res = mod(**inputs) 2025-10-10T01:00:36.8100034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8100422Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8100824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8101235Z outputs = self.model.decoder( 2025-10-10T01:00:36.8101612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8102016Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8102706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8103115Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8103507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8103904Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8104324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8104725Z return func(*args, **kwargs) 2025-10-10T01:00:36.8105122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8105571Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8106034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8106542Z return func(*args, **kwargs) 2025-10-10T01:00:36.8106948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8107387Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8107555Z 2025-10-10T01:00:36.8107649Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8107894Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8108175Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8108560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8108913Z res = mod(**inputs) 2025-10-10T01:00:36.8109276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8109670Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8110086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8110500Z outputs = self.model.decoder( 2025-10-10T01:00:36.8110876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8111264Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8111671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8112070Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8112453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8112851Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8113268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8113669Z return func(*args, **kwargs) 2025-10-10T01:00:36.8114062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8114495Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8114927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8115331Z return func(*args, **kwargs) 2025-10-10T01:00:36.8115714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8116151Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8116635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8117159Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8117386Z 2025-10-10T01:00:36.8117511Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8117915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8118268Z res = mod(**inputs) 2025-10-10T01:00:36.8118623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8119011Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8119412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8119822Z outputs = self.model.decoder( 2025-10-10T01:00:36.8120196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8120577Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8120984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8121418Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8121802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8122203Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8122626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8123100Z return func(*args, **kwargs) 2025-10-10T01:00:36.8123518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8123976Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8124435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8124844Z return func(*args, **kwargs) 2025-10-10T01:00:36.8125243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8125680Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8125845Z 2025-10-10T01:00:36.8125966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8126378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8126758Z res = mod(**inputs) 2025-10-10T01:00:36.8127133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8127546Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8127989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8128418Z outputs = self.model.decoder( 2025-10-10T01:00:36.8128806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8129214Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8129642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8130068Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8130468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8130875Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8131309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8131914Z return func(*args, **kwargs) 2025-10-10T01:00:36.8132322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8132763Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8132918Z 2025-10-10T01:00:36.8133102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8133526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8133878Z res = mod(**inputs) 2025-10-10T01:00:36.8134211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8134563Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8134950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8135336Z outputs = self.model.decoder( 2025-10-10T01:00:36.8135691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8136054Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8136426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8136867Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8137231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8137602Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8137984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8138367Z return func(*args, **kwargs) 2025-10-10T01:00:36.8138735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8139145Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8139302Z 2025-10-10T01:00:36.8139418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8139775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8140109Z res = mod(**inputs) 2025-10-10T01:00:36.8140448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8140831Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8141237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8141635Z outputs = self.model.decoder( 2025-10-10T01:00:36.8142016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8142407Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8142822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8143230Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8143621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8144032Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8144459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8144876Z return func(*args, **kwargs) 2025-10-10T01:00:36.8145270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8145698Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8145860Z 2025-10-10T01:00:36.8145978Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8146432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8146801Z res = mod(**inputs) 2025-10-10T01:00:36.8147163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8147562Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8148016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8148403Z outputs = self.model.decoder( 2025-10-10T01:00:36.8148750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8149115Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8149499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8149895Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8150239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8150583Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8150950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8151336Z return func(*args, **kwargs) 2025-10-10T01:00:36.8151706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-10-10T01:00:36.8152134Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-10-10T01:00:36.8152331Z 2025-10-10T01:00:36.8152437Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8152805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8153133Z res = mod(**inputs) 2025-10-10T01:00:36.8153468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8153829Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8154201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8154577Z outputs = self.model.decoder( 2025-10-10T01:00:36.8154928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8155281Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8155647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8156019Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8156365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8156725Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8157102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8157483Z return func(*args, **kwargs) 2025-10-10T01:00:36.8157854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8158269Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8158683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8159044Z return func(*args, **kwargs) 2025-10-10T01:00:36.8159404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8159812Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8159975Z 2025-10-10T01:00:36.8160087Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8160441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8160759Z res = mod(**inputs) 2025-10-10T01:00:36.8161084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8161435Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8161882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8162248Z outputs = self.model.decoder( 2025-10-10T01:00:36.8162589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8162940Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8163308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8163678Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8164018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8164373Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8164748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8165145Z return func(*args, **kwargs) 2025-10-10T01:00:36.8165494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8165883Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8166266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8166624Z return func(*args, **kwargs) 2025-10-10T01:00:36.8166974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8167336Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8167478Z 2025-10-10T01:00:36.8167577Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8167926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8168238Z res = mod(**inputs) 2025-10-10T01:00:36.8168559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8168897Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8169261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8169627Z outputs = self.model.decoder( 2025-10-10T01:00:36.8169969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8170327Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8170689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8171054Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8171394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8171755Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8172126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8172506Z return func(*args, **kwargs) 2025-10-10T01:00:36.8172875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8173286Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8173688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8174067Z return func(*args, **kwargs) 2025-10-10T01:00:36.8174420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8174802Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8174938Z 2025-10-10T01:00:36.8175050Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8175262Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8175523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8175873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8176184Z res = mod(**inputs) 2025-10-10T01:00:36.8176495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8176838Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8177205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8177569Z outputs = self.model.decoder( 2025-10-10T01:00:36.8177902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8178238Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8178636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8179004Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8179352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8179725Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8180107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8180486Z return func(*args, **kwargs) 2025-10-10T01:00:36.8180857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8181274Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8181664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8182042Z return func(*args, **kwargs) 2025-10-10T01:00:36.8182411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8182824Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8183307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8183821Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8184025Z 2025-10-10T01:00:36.8184142Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8184510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8184850Z res = mod(**inputs) 2025-10-10T01:00:36.8185209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8185595Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8186015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8186501Z outputs = self.model.decoder( 2025-10-10T01:00:36.8186883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8187281Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8187714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8188130Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8188484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8188850Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8189228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8189627Z return func(*args, **kwargs) 2025-10-10T01:00:36.8190007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8190411Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8190811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8191181Z return func(*args, **kwargs) 2025-10-10T01:00:36.8191548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8191942Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8192079Z 2025-10-10T01:00:36.8192191Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8192545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8192887Z res = mod(**inputs) 2025-10-10T01:00:36.8193234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8193588Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8193964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8194339Z outputs = self.model.decoder( 2025-10-10T01:00:36.8194692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8195067Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8195436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8195799Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8196150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8196515Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8196894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8197262Z return func(*args, **kwargs) 2025-10-10T01:00:36.8197614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8197997Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8198140Z 2025-10-10T01:00:36.8198243Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8198601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8198921Z res = mod(**inputs) 2025-10-10T01:00:36.8199239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8199592Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8199971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8200341Z outputs = self.model.decoder( 2025-10-10T01:00:36.8200684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8201076Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8201458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8201846Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8202196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8202560Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8202947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8203355Z return func(*args, **kwargs) 2025-10-10T01:00:36.8203760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8204185Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8204360Z 2025-10-10T01:00:36.8204472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8204860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8205209Z res = mod(**inputs) 2025-10-10T01:00:36.8205561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8205937Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8206350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8206772Z outputs = self.model.decoder( 2025-10-10T01:00:36.8207211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8207595Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8207990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8208394Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8208775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8209164Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8209568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8209973Z return func(*args, **kwargs) 2025-10-10T01:00:36.8210365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8210761Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8210901Z 2025-10-10T01:00:36.8211018Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8211374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8211699Z res = mod(**inputs) 2025-10-10T01:00:36.8212035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8212419Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8212817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8213221Z outputs = self.model.decoder( 2025-10-10T01:00:36.8213592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8213977Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8214386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8214777Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8215156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8215544Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8215950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8216352Z return func(*args, **kwargs) 2025-10-10T01:00:36.8216732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8217159Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8217585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8218021Z return func(*args, **kwargs) 2025-10-10T01:00:36.8218302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8218434Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8218437Z 2025-10-10T01:00:36.8218550Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8218763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8218841Z res = mod(**inputs) 2025-10-10T01:00:36.8219080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8219170Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8219432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8219512Z outputs = self.model.decoder( 2025-10-10T01:00:36.8219792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8219876Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8220139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8220216Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8220455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8220550Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8220809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8220893Z return func(*args, **kwargs) 2025-10-10T01:00:36.8221158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8221281Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8221546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8221624Z return func(*args, **kwargs) 2025-10-10T01:00:36.8221898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8221991Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8221995Z 2025-10-10T01:00:36.8222117Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8222349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8222423Z res = mod(**inputs) 2025-10-10T01:00:36.8222681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8222767Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8223046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8223129Z outputs = self.model.decoder( 2025-10-10T01:00:36.8223376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8223470Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8223741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8223830Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8224069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8224168Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8224440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8224579Z return func(*args, **kwargs) 2025-10-10T01:00:36.8224877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8224990Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8225268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8225346Z return func(*args, **kwargs) 2025-10-10T01:00:36.8225621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8225726Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8225730Z 2025-10-10T01:00:36.8225821Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8225920Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8226035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8226269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8226450Z res = mod(**inputs) 2025-10-10T01:00:36.8226705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8226799Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8227081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8227176Z outputs = self.model.decoder( 2025-10-10T01:00:36.8227419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8227504Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8227788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8227867Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8228126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8228224Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8228488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8228575Z return func(*args, **kwargs) 2025-10-10T01:00:36.8228837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8228953Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8229220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8229296Z return func(*args, **kwargs) 2025-10-10T01:00:36.8229571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8229686Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8230018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8230168Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8230172Z 2025-10-10T01:00:36.8230292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8230513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8230586Z res = mod(**inputs) 2025-10-10T01:00:36.8230838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8230920Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8231191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8231293Z outputs = self.model.decoder( 2025-10-10T01:00:36.8231716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8231813Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8232077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8232164Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8232407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8232504Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8232764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8232842Z return func(*args, **kwargs) 2025-10-10T01:00:36.8233109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8233272Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8233543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8233619Z return func(*args, **kwargs) 2025-10-10T01:00:36.8233878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8233981Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8233985Z 2025-10-10T01:00:36.8234097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8234316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8234389Z res = mod(**inputs) 2025-10-10T01:00:36.8234626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8234716Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8234978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8235068Z outputs = self.model.decoder( 2025-10-10T01:00:36.8235303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8235390Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8235650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8235730Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8235978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8236064Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8236328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8236410Z return func(*args, **kwargs) 2025-10-10T01:00:36.8236670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8236769Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8236773Z 2025-10-10T01:00:36.8236884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8237107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8237177Z res = mod(**inputs) 2025-10-10T01:00:36.8237415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8237505Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8237762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8237850Z outputs = self.model.decoder( 2025-10-10T01:00:36.8238133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8238222Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8238485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8238563Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8238808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8238895Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8239163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8239239Z return func(*args, **kwargs) 2025-10-10T01:00:36.8239495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8239661Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8239666Z 2025-10-10T01:00:36.8239780Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8240000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8240073Z res = mod(**inputs) 2025-10-10T01:00:36.8240312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8240400Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8240660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8240748Z outputs = self.model.decoder( 2025-10-10T01:00:36.8240981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8241071Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8241336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8241416Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8241658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8241753Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8242020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8242096Z return func(*args, **kwargs) 2025-10-10T01:00:36.8242352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8242449Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8242453Z 2025-10-10T01:00:36.8242563Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8242784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8242860Z res = mod(**inputs) 2025-10-10T01:00:36.8243096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8243185Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8243448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8243531Z outputs = self.model.decoder( 2025-10-10T01:00:36.8243754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8243837Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8244085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8244159Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8244415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8244520Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8244778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8244849Z return func(*args, **kwargs) 2025-10-10T01:00:36.8245097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-10-10T01:00:36.8245241Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-10-10T01:00:36.8245245Z 2025-10-10T01:00:36.8245350Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8245557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8245625Z res = mod(**inputs) 2025-10-10T01:00:36.8245873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8245972Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8246219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8246302Z outputs = self.model.decoder( 2025-10-10T01:00:36.8246521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8246605Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8246852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8246927Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8247159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8247239Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8247497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8247569Z return func(*args, **kwargs) 2025-10-10T01:00:36.8247812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8247920Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8248164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8248243Z return func(*args, **kwargs) 2025-10-10T01:00:36.8248491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8248613Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8248616Z 2025-10-10T01:00:36.8248722Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8248927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8249004Z res = mod(**inputs) 2025-10-10T01:00:36.8249225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8249308Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8249554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8249629Z outputs = self.model.decoder( 2025-10-10T01:00:36.8249858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8249936Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8250190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8250266Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8250531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8250621Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8250867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8250946Z return func(*args, **kwargs) 2025-10-10T01:00:36.8251196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8251304Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8251551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8251622Z return func(*args, **kwargs) 2025-10-10T01:00:36.8251877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8252000Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8252005Z 2025-10-10T01:00:36.8252120Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8252319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8252387Z res = mod(**inputs) 2025-10-10T01:00:36.8252617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8252694Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8252946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8253022Z outputs = self.model.decoder( 2025-10-10T01:00:36.8253244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8253329Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8253577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8253659Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8253885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8253976Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8254222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8254294Z return func(*args, **kwargs) 2025-10-10T01:00:36.8254544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8254644Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8254895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8254971Z return func(*args, **kwargs) 2025-10-10T01:00:36.8255216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8255313Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8255317Z 2025-10-10T01:00:36.8255402Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8255492Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8255598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8255797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8255872Z res = mod(**inputs) 2025-10-10T01:00:36.8256094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8256179Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8256424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8256546Z outputs = self.model.decoder( 2025-10-10T01:00:36.8256768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8256845Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8257100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8257175Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8257407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8257489Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8257733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8257815Z return func(*args, **kwargs) 2025-10-10T01:00:36.8258083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8258215Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8258459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8258530Z return func(*args, **kwargs) 2025-10-10T01:00:36.8258781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8258884Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8259189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8259330Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8259333Z 2025-10-10T01:00:36.8259446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8259653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8259720Z res = mod(**inputs) 2025-10-10T01:00:36.8259951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8260030Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8260285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8260360Z outputs = self.model.decoder( 2025-10-10T01:00:36.8260581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8260666Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8260909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8260991Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8261222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8261314Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8261572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8261649Z return func(*args, **kwargs) 2025-10-10T01:00:36.8261912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8262018Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8262280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8262356Z return func(*args, **kwargs) 2025-10-10T01:00:36.8262611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8262734Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8262754Z 2025-10-10T01:00:36.8262866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8263084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8263154Z res = mod(**inputs) 2025-10-10T01:00:36.8263394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8263481Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8263727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8263810Z outputs = self.model.decoder( 2025-10-10T01:00:36.8264032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8264115Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8264409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8264489Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8264733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8264819Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8265086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8265162Z return func(*args, **kwargs) 2025-10-10T01:00:36.8265419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8265517Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8265521Z 2025-10-10T01:00:36.8265632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8265855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8265928Z res = mod(**inputs) 2025-10-10T01:00:36.8266161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8266257Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8266603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8266700Z outputs = self.model.decoder( 2025-10-10T01:00:36.8266951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8267044Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8267318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8267399Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8267669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8267757Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8268027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8268103Z return func(*args, **kwargs) 2025-10-10T01:00:36.8268364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8268476Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8268480Z 2025-10-10T01:00:36.8268586Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8268798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8268865Z res = mod(**inputs) 2025-10-10T01:00:36.8269099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8269221Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8269470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8269554Z outputs = self.model.decoder( 2025-10-10T01:00:36.8269778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8269863Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8270113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8270188Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8270424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8270506Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8270796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8270873Z return func(*args, **kwargs) 2025-10-10T01:00:36.8271134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8271232Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8271236Z 2025-10-10T01:00:36.8271347Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8271568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8271639Z res = mod(**inputs) 2025-10-10T01:00:36.8271882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8271962Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8272225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8272319Z outputs = self.model.decoder( 2025-10-10T01:00:36.8272554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8272641Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8272904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8272982Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8273229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8273316Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8273582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8273659Z return func(*args, **kwargs) 2025-10-10T01:00:36.8273922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8274040Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8274299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8274383Z return func(*args, **kwargs) 2025-10-10T01:00:36.8274643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8274781Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8274785Z 2025-10-10T01:00:36.8274890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8275092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8275166Z res = mod(**inputs) 2025-10-10T01:00:36.8275390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8275511Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8275761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8275835Z outputs = self.model.decoder( 2025-10-10T01:00:36.8276066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8276144Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8276399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8276473Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8276699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8276790Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8277056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8277151Z return func(*args, **kwargs) 2025-10-10T01:00:36.8277395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8277503Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8277745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8277815Z return func(*args, **kwargs) 2025-10-10T01:00:36.8278064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8278145Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8278149Z 2025-10-10T01:00:36.8278259Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8278458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8278528Z res = mod(**inputs) 2025-10-10T01:00:36.8278754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8278831Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8279081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8279156Z outputs = self.model.decoder( 2025-10-10T01:00:36.8279376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8279459Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8279700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8279782Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8280010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8280103Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8280350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8280426Z return func(*args, **kwargs) 2025-10-10T01:00:36.8280690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8280796Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8281061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8281136Z return func(*args, **kwargs) 2025-10-10T01:00:36.8281396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8281523Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8281527Z 2025-10-10T01:00:36.8281634Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8281731Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8281842Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8282055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8282133Z res = mod(**inputs) 2025-10-10T01:00:36.8282367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8282451Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8282697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8282781Z outputs = self.model.decoder( 2025-10-10T01:00:36.8283005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8283119Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8283373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8283449Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8283684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8283766Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8284010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8284090Z return func(*args, **kwargs) 2025-10-10T01:00:36.8284331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8284438Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8284686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8284760Z return func(*args, **kwargs) 2025-10-10T01:00:36.8285010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8285112Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8285415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8285554Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8285559Z 2025-10-10T01:00:36.8285670Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8285877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8285948Z res = mod(**inputs) 2025-10-10T01:00:36.8286192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8286277Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8286540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8286619Z outputs = self.model.decoder( 2025-10-10T01:00:36.8286855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8286941Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8287197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8287284Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8287523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8287616Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8287927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8288004Z return func(*args, **kwargs) 2025-10-10T01:00:36.8288273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8288378Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8288645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8288720Z return func(*args, **kwargs) 2025-10-10T01:00:36.8288979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8289076Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8289080Z 2025-10-10T01:00:36.8289192Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8289451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8289526Z res = mod(**inputs) 2025-10-10T01:00:36.8289759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8289848Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8290107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8290193Z outputs = self.model.decoder( 2025-10-10T01:00:36.8290430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8290519Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8290782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8290860Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8291118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8291205Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8291475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8291551Z return func(*args, **kwargs) 2025-10-10T01:00:36.8291812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8291908Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8291911Z 2025-10-10T01:00:36.8292022Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8292249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8292321Z res = mod(**inputs) 2025-10-10T01:00:36.8292562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8292653Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8292914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8293001Z outputs = self.model.decoder( 2025-10-10T01:00:36.8293242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8293329Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8293594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8293671Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8293919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8294005Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8294309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8294387Z return func(*args, **kwargs) 2025-10-10T01:00:36.8294649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8294763Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8294767Z 2025-10-10T01:00:36.8294877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8295096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8295165Z res = mod(**inputs) 2025-10-10T01:00:36.8295402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8295489Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8295765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8295875Z outputs = self.model.decoder( 2025-10-10T01:00:36.8296112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8296201Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8296465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8296544Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8296794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8296881Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8297152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8297228Z return func(*args, **kwargs) 2025-10-10T01:00:36.8297494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8297593Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8297597Z 2025-10-10T01:00:36.8297709Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8297928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8298004Z res = mod(**inputs) 2025-10-10T01:00:36.8298241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8298330Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8298593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8298683Z outputs = self.model.decoder( 2025-10-10T01:00:36.8298922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8299014Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8299280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8299359Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8299610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8299698Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8299968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8300044Z return func(*args, **kwargs) 2025-10-10T01:00:36.8300306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-10-10T01:00:36.8300466Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-10-10T01:00:36.8300490Z 2025-10-10T01:00:36.8300607Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8300852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8300931Z res = mod(**inputs) 2025-10-10T01:00:36.8301183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8301269Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8301536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8301626Z outputs = self.model.decoder( 2025-10-10T01:00:36.8301868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8301959Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8302248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8302349Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8302603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8302692Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8302967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8303046Z return func(*args, **kwargs) 2025-10-10T01:00:36.8303314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8303434Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8303699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8303785Z return func(*args, **kwargs) 2025-10-10T01:00:36.8304056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8304191Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8304195Z 2025-10-10T01:00:36.8304308Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8304526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8304607Z res = mod(**inputs) 2025-10-10T01:00:36.8304850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8304942Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8305206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8305288Z outputs = self.model.decoder( 2025-10-10T01:00:36.8305538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8305628Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8305913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8305995Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8306238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8306413Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8306699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8306787Z return func(*args, **kwargs) 2025-10-10T01:00:36.8307067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8307186Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8307503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8307584Z return func(*args, **kwargs) 2025-10-10T01:00:36.8307856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8307948Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8307953Z 2025-10-10T01:00:36.8308085Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8308298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8308368Z res = mod(**inputs) 2025-10-10T01:00:36.8308612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8308694Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8308981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8309087Z outputs = self.model.decoder( 2025-10-10T01:00:36.8309331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8309412Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8309674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8309759Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8310006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8310104Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8310385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8310462Z return func(*args, **kwargs) 2025-10-10T01:00:36.8310762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8310870Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8311147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8311224Z return func(*args, **kwargs) 2025-10-10T01:00:36.8311498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8311601Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8311605Z 2025-10-10T01:00:36.8311695Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8311801Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8311912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8312124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8312206Z res = mod(**inputs) 2025-10-10T01:00:36.8312446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8312538Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8312815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8312907Z outputs = self.model.decoder( 2025-10-10T01:00:36.8313155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8313240Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8313514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8313595Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8313848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8313972Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8314274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8314363Z return func(*args, **kwargs) 2025-10-10T01:00:36.8314639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8314756Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8315029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8315106Z return func(*args, **kwargs) 2025-10-10T01:00:36.8315386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8315495Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8315845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8316017Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8316021Z 2025-10-10T01:00:36.8316143Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8316363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8316435Z res = mod(**inputs) 2025-10-10T01:00:36.8316686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8316772Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8317048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8317130Z outputs = self.model.decoder( 2025-10-10T01:00:36.8317371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8317466Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8317731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8317819Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8318065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8318160Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8318426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8318504Z return func(*args, **kwargs) 2025-10-10T01:00:36.8318776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8318884Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8319164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8319243Z return func(*args, **kwargs) 2025-10-10T01:00:36.8319508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8319608Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8319611Z 2025-10-10T01:00:36.8319725Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8319952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8320024Z res = mod(**inputs) 2025-10-10T01:00:36.8320268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8320360Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8320631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8320759Z outputs = self.model.decoder( 2025-10-10T01:00:36.8321003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8321094Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8321359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8321441Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8321692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8321781Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8322055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8322131Z return func(*args, **kwargs) 2025-10-10T01:00:36.8322473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8322575Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8322579Z 2025-10-10T01:00:36.8322691Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8322916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8322988Z res = mod(**inputs) 2025-10-10T01:00:36.8323226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8323315Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8323579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8323668Z outputs = self.model.decoder( 2025-10-10T01:00:36.8323909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8324005Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8324269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8324348Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8324602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8324689Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8324960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8325035Z return func(*args, **kwargs) 2025-10-10T01:00:36.8325299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8325415Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8325422Z 2025-10-10T01:00:36.8325536Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8325764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8325838Z res = mod(**inputs) 2025-10-10T01:00:36.8326096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8326175Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8326433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8326518Z outputs = self.model.decoder( 2025-10-10T01:00:36.8326756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8326844Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8327104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8327220Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8327466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8327551Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8327816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8327891Z return func(*args, **kwargs) 2025-10-10T01:00:36.8328150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8328248Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8328252Z 2025-10-10T01:00:36.8328361Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8328581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8328689Z res = mod(**inputs) 2025-10-10T01:00:36.8328933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8329014Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8329272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8329360Z outputs = self.model.decoder( 2025-10-10T01:00:36.8329593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8329683Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8329942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8330021Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8330267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8330358Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8330628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8330704Z return func(*args, **kwargs) 2025-10-10T01:00:36.8330963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8331075Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8331334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8331418Z return func(*args, **kwargs) 2025-10-10T01:00:36.8331838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8331975Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8331983Z 2025-10-10T01:00:36.8332098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8332316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8332397Z res = mod(**inputs) 2025-10-10T01:00:36.8332635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8332724Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8332987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8333069Z outputs = self.model.decoder( 2025-10-10T01:00:36.8333316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8333398Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8333672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8333807Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8334083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8334178Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8334440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8334523Z return func(*args, **kwargs) 2025-10-10T01:00:36.8334784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8334896Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8335155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8335230Z return func(*args, **kwargs) 2025-10-10T01:00:36.8335534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8335661Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8335665Z 2025-10-10T01:00:36.8335783Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8335994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8336065Z res = mod(**inputs) 2025-10-10T01:00:36.8336308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8336391Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8336661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8336735Z outputs = self.model.decoder( 2025-10-10T01:00:36.8336949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8337037Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8337277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8337358Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8337577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8337664Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8337902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8337972Z return func(*args, **kwargs) 2025-10-10T01:00:36.8338214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8338312Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8338560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8338633Z return func(*args, **kwargs) 2025-10-10T01:00:36.8338870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8338965Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8338969Z 2025-10-10T01:00:36.8339052Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8339140Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8339241Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8339434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8339504Z res = mod(**inputs) 2025-10-10T01:00:36.8339719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8339817Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8340071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8340153Z outputs = self.model.decoder( 2025-10-10T01:00:36.8340367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8340444Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8340697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8340771Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8341004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8341090Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8341349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8341467Z return func(*args, **kwargs) 2025-10-10T01:00:36.8341727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8341841Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8342097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8342172Z return func(*args, **kwargs) 2025-10-10T01:00:36.8342436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8342542Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8342864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8343012Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8343020Z 2025-10-10T01:00:36.8343140Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8343352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8343423Z res = mod(**inputs) 2025-10-10T01:00:36.8343664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8343746Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8344010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8344089Z outputs = self.model.decoder( 2025-10-10T01:00:36.8344320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8344408Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8344670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8344761Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8344999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8345092Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8345350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8345427Z return func(*args, **kwargs) 2025-10-10T01:00:36.8345699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8345809Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8346082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8346159Z return func(*args, **kwargs) 2025-10-10T01:00:36.8346529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8346637Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8346642Z 2025-10-10T01:00:36.8346756Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8346985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8347057Z res = mod(**inputs) 2025-10-10T01:00:36.8347349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8347435Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8347689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8347773Z outputs = self.model.decoder( 2025-10-10T01:00:36.8348017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8348122Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8348373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8348448Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8348675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8348755Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8349001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8349070Z return func(*args, **kwargs) 2025-10-10T01:00:36.8349310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8349400Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8349407Z 2025-10-10T01:00:36.8349510Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8349711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8349775Z res = mod(**inputs) 2025-10-10T01:00:36.8349991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8350076Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8350313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8350394Z outputs = self.model.decoder( 2025-10-10T01:00:36.8350609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8350691Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8350939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8351019Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8351250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8351331Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8351582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8351653Z return func(*args, **kwargs) 2025-10-10T01:00:36.8351899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8352014Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8352018Z 2025-10-10T01:00:36.8352129Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8352347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8352442Z res = mod(**inputs) 2025-10-10T01:00:36.8352690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8352775Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8353018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8353097Z outputs = self.model.decoder( 2025-10-10T01:00:36.8353313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8353394Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8353634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8353706Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8353933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8354054Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8354307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8354376Z return func(*args, **kwargs) 2025-10-10T01:00:36.8354623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8354713Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8354717Z 2025-10-10T01:00:36.8354820Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8355029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8355095Z res = mod(**inputs) 2025-10-10T01:00:36.8355325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8355404Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8355653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8355736Z outputs = self.model.decoder( 2025-10-10T01:00:36.8355957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8356039Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8356283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8356358Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8356590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8356672Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8356923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8356999Z return func(*args, **kwargs) 2025-10-10T01:00:36.8357245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-10-10T01:00:36.8357391Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-10-10T01:00:36.8357394Z 2025-10-10T01:00:36.8357498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8357706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8357772Z res = mod(**inputs) 2025-10-10T01:00:36.8358003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8358080Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8358325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8358428Z outputs = self.model.decoder( 2025-10-10T01:00:36.8358665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8358749Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8358992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8359066Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8359300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8359382Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8359633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8359705Z return func(*args, **kwargs) 2025-10-10T01:00:36.8359949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8360446Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8360694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8360773Z return func(*args, **kwargs) 2025-10-10T01:00:36.8361017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-10-10T01:00:36.8361141Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:00:36.8361144Z 2025-10-10T01:00:36.8361249Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8361451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8361528Z res = mod(**inputs) 2025-10-10T01:00:36.8361748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8361837Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8362090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8362167Z outputs = self.model.decoder( 2025-10-10T01:00:36.8362401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8362477Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8362729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8362805Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8363031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8363122Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8363380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8363469Z return func(*args, **kwargs) 2025-10-10T01:00:36.8363728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8363849Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8364090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8364161Z return func(*args, **kwargs) 2025-10-10T01:00:36.8364413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-10-10T01:00:36.8364496Z key_states = self.k_proj(hidden_states) 2025-10-10T01:00:36.8364499Z 2025-10-10T01:00:36.8364609Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8364809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8364900Z res = mod(**inputs) 2025-10-10T01:00:36.8365148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8365228Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8365480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8365556Z outputs = self.model.decoder( 2025-10-10T01:00:36.8365786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8365863Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8366107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8366189Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8366416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8366541Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8366788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8366859Z return func(*args, **kwargs) 2025-10-10T01:00:36.8367112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8367210Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8367463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8367533Z return func(*args, **kwargs) 2025-10-10T01:00:36.8367776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-10-10T01:00:36.8367872Z value_states = self.v_proj(hidden_states) 2025-10-10T01:00:36.8367877Z 2025-10-10T01:00:36.8367962Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8368054Z cudagraph partition due to non gpu ops 2025-10-10T01:00:36.8368157Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8368358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8368431Z res = mod(**inputs) 2025-10-10T01:00:36.8368653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8368736Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8368982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8369065Z outputs = self.model.decoder( 2025-10-10T01:00:36.8369295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8399827Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8400387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8400486Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8400736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8400837Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8401099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8401180Z return func(*args, **kwargs) 2025-10-10T01:00:36.8401448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8401558Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8401818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8402000Z return func(*args, **kwargs) 2025-10-10T01:00:36.8402295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-10-10T01:00:36.8402405Z attn_output, attn_weights = attention_interface( 2025-10-10T01:00:36.8402712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:00:36.8402862Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:00:36.8402869Z 2025-10-10T01:00:36.8402991Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8403227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8403304Z res = mod(**inputs) 2025-10-10T01:00:36.8403553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8403724Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8403998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8404096Z outputs = self.model.decoder( 2025-10-10T01:00:36.8404339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8404434Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8404702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8404786Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8405041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8405134Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8405412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8405499Z return func(*args, **kwargs) 2025-10-10T01:00:36.8405769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-10-10T01:00:36.8405886Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:00:36.8406140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8406223Z return func(*args, **kwargs) 2025-10-10T01:00:36.8406476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-10-10T01:00:36.8406562Z attn_output = self.out_proj(attn_output) 2025-10-10T01:00:36.8406575Z 2025-10-10T01:00:36.8406687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8406897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8406981Z res = mod(**inputs) 2025-10-10T01:00:36.8407212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8407301Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8407553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8407634Z outputs = self.model.decoder( 2025-10-10T01:00:36.8407869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8407947Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8408215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8408297Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8408538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8408671Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8408935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8409018Z return func(*args, **kwargs) 2025-10-10T01:00:36.8409279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-10-10T01:00:36.8409374Z hidden_states = self.fc1(hidden_states) 2025-10-10T01:00:36.8409385Z 2025-10-10T01:00:36.8409501Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8409720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8409802Z res = mod(**inputs) 2025-10-10T01:00:36.8410040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8410170Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8410441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8410522Z outputs = self.model.decoder( 2025-10-10T01:00:36.8410766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8410848Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8411119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8411200Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8411440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8411535Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8411801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8411889Z return func(*args, **kwargs) 2025-10-10T01:00:36.8412156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-10-10T01:00:36.8412273Z hidden_states = self.activation_fn(hidden_states) 2025-10-10T01:00:36.8412277Z 2025-10-10T01:00:36.8412388Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8412608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8412691Z res = mod(**inputs) 2025-10-10T01:00:36.8412929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8413021Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8413286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-10-10T01:00:36.8413373Z outputs = self.model.decoder( 2025-10-10T01:00:36.8413617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8413698Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8413970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-10-10T01:00:36.8414052Z layer_outputs = decoder_layer( 2025-10-10T01:00:36.8414291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:00:36.8414387Z return super().__call__(*args, **kwargs) 2025-10-10T01:00:36.8414651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:00:36.8414735Z return func(*args, **kwargs) 2025-10-10T01:00:36.8414998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-10-10T01:00:36.8415117Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:00:36.8415138Z 2025-10-10T01:00:36.8415252Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8415465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8415548Z res = mod(**inputs) 2025-10-10T01:00:36.8415785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8415876Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8416135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-10-10T01:00:36.8416240Z logits = self.lm_head(outputs[0]).contiguous() 2025-10-10T01:00:36.8416244Z 2025-10-10T01:00:36.8416365Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:00:36.8416595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:00:36.8416708Z res = mod(**inputs) 2025-10-10T01:00:36.8416943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-10-10T01:00:36.8417026Z output = func(self, *args, **kwargs) 2025-10-10T01:00:36.8417294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-10-10T01:00:36.8417380Z loss = self.loss_function( 2025-10-10T01:00:36.8417648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T01:00:36.8417836Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T01:00:36.8418116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T01:00:36.8418327Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T01:00:36.8418335Z 2025-10-10T01:00:48.3100096Z Compilation time (from dynamo_timed): 17.029196355 2025-10-10T01:00:48.3657901Z pass 2025-10-10T01:00:48.3660041Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:48.3660859Z TIMING: _recursive_pre_grad_passes:0.00878 _recursive_joint_graph_passes:0.70821 _recursive_post_grad_passes:0.09027 async_compile.wait:0.88484 code_gen:10.12852 inductor_compile:11.42794 backend_compile:14.65163 gc:0.00034 entire_frame_compile:17.0292 total_wall_time:17.0292 2025-10-10T01:00:48.3661833Z STATS: call_* op count: 379 | FakeTensorMode.__torch_dispatch__:7117 | FakeTensor.__torch_dispatch__:4035 | ProxyTorchDispatchMode.__torch_dispatch__:1929 2025-10-10T01:00:48.3662327Z Dynamo produced 1 graphs covering 379 ops with 0 graph breaks (0 unique) 2025-10-10T01:00:51.1347751Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:00:51.1348637Z import pynvml # type: ignore[import] 2025-10-10T01:00:54.6060198Z 2025-10-10T01:00:55.8274151Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:00:55.8274598Z loading model: 0it [00:01, ?it/s] 2025-10-10T01:00:55.8290180Z cpu eval PLBartForCausalLM 2025-10-10T01:00:56.5124658Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:56.8179934Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:00:57.1206434Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:02.2507492Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2508252Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2508508Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2508813Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2509049Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2509286Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2509563Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2510002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2510383Z res = mod(**inputs) 2025-10-10T01:01:02.2510915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2511392Z outputs = self.model.decoder( 2025-10-10T01:01:02.2511904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2512365Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2512880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2513303Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2513757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2514200Z return func(*args, **kwargs) 2025-10-10T01:01:02.2514659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2515154Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2515632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2516056Z return func(*args, **kwargs) 2025-10-10T01:01:02.2516501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2517053Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2517288Z 2025-10-10T01:01:02.2517413Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2517820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2518196Z res = mod(**inputs) 2025-10-10T01:01:02.2518625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2519077Z outputs = self.model.decoder( 2025-10-10T01:01:02.2519528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2519970Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2520370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2520793Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2521228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2521656Z return func(*args, **kwargs) 2025-10-10T01:01:02.2522084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2522564Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2523019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2523447Z return func(*args, **kwargs) 2025-10-10T01:01:02.2523878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2524340Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2524496Z 2025-10-10T01:01:02.2524654Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2525077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2525556Z res = mod(**inputs) 2025-10-10T01:01:02.2525973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2526426Z outputs = self.model.decoder( 2025-10-10T01:01:02.2526866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2527320Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2527749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2528171Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2528586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2529063Z return func(*args, **kwargs) 2025-10-10T01:01:02.2529501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2529998Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2530447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2530875Z return func(*args, **kwargs) 2025-10-10T01:01:02.2531298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2531951Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2532114Z 2025-10-10T01:01:02.2532216Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2532500Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2532908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2533286Z res = mod(**inputs) 2025-10-10T01:01:02.2533681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2534118Z outputs = self.model.decoder( 2025-10-10T01:01:02.2534557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2534989Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2535367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2535750Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2536160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2536564Z return func(*args, **kwargs) 2025-10-10T01:01:02.2536984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2537433Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2537856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2538264Z return func(*args, **kwargs) 2025-10-10T01:01:02.2538673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2539131Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2539608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2540129Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2540336Z 2025-10-10T01:01:02.2540508Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2540930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2541278Z res = mod(**inputs) 2025-10-10T01:01:02.2541692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2542130Z outputs = self.model.decoder( 2025-10-10T01:01:02.2542548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2542982Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2543358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2543740Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2544152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2544613Z return func(*args, **kwargs) 2025-10-10T01:01:02.2545027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2545475Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2545897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2546299Z return func(*args, **kwargs) 2025-10-10T01:01:02.2546782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2547228Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2547385Z 2025-10-10T01:01:02.2547504Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2547905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2548271Z res = mod(**inputs) 2025-10-10T01:01:02.2548688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2549115Z outputs = self.model.decoder( 2025-10-10T01:01:02.2549526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2549951Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2550333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2550729Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2551132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2551541Z return func(*args, **kwargs) 2025-10-10T01:01:02.2551930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2552396Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2552590Z 2025-10-10T01:01:02.2552712Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2553087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2553417Z res = mod(**inputs) 2025-10-10T01:01:02.2553792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2554194Z outputs = self.model.decoder( 2025-10-10T01:01:02.2554583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2554973Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2555334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2555733Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2556149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2556537Z return func(*args, **kwargs) 2025-10-10T01:01:02.2556922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2557370Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2557774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2558133Z return self.act(input) 2025-10-10T01:01:02.2558250Z 2025-10-10T01:01:02.2558359Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2558729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2559057Z res = mod(**inputs) 2025-10-10T01:01:02.2559502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2559908Z outputs = self.model.decoder( 2025-10-10T01:01:02.2560293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2560694Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2561051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2561420Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2561803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2562184Z return func(*args, **kwargs) 2025-10-10T01:01:02.2562573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2562990Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2563135Z 2025-10-10T01:01:02.2563248Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2563630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2563992Z res = mod(**inputs) 2025-10-10T01:01:02.2564370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2564771Z outputs = self.model.decoder( 2025-10-10T01:01:02.2565166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2565561Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2565921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2566293Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2566693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2567065Z return func(*args, **kwargs) 2025-10-10T01:01:02.2567456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2567884Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2568293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2568672Z return func(*args, **kwargs) 2025-10-10T01:01:02.2569053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2569533Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2569747Z 2025-10-10T01:01:02.2569880Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2570268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2570599Z res = mod(**inputs) 2025-10-10T01:01:02.2570971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2571380Z outputs = self.model.decoder( 2025-10-10T01:01:02.2571778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2572183Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2572541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2572939Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2573361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2573826Z return func(*args, **kwargs) 2025-10-10T01:01:02.2574251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2574697Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2575130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2575509Z return func(*args, **kwargs) 2025-10-10T01:01:02.2575894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2576310Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2576457Z 2025-10-10T01:01:02.2576569Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2576953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2577306Z res = mod(**inputs) 2025-10-10T01:01:02.2577721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2578134Z outputs = self.model.decoder( 2025-10-10T01:01:02.2578550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2578979Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2579360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2579752Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2580156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2580569Z return func(*args, **kwargs) 2025-10-10T01:01:02.2580983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2581440Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2581870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2582270Z return func(*args, **kwargs) 2025-10-10T01:01:02.2582679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2583125Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2583280Z 2025-10-10T01:01:02.2583378Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2583633Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2584018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2584371Z res = mod(**inputs) 2025-10-10T01:01:02.2584775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2585251Z outputs = self.model.decoder( 2025-10-10T01:01:02.2585667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2586097Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2586591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2587006Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2587422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2587837Z return func(*args, **kwargs) 2025-10-10T01:01:02.2588274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2588731Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2589214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2589609Z return func(*args, **kwargs) 2025-10-10T01:01:02.2590023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2590478Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2590960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2591485Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2591684Z 2025-10-10T01:01:02.2591799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2592186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2592536Z res = mod(**inputs) 2025-10-10T01:01:02.2592943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2593371Z outputs = self.model.decoder( 2025-10-10T01:01:02.2593780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2594201Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2594580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2594969Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2595372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2595772Z return func(*args, **kwargs) 2025-10-10T01:01:02.2596182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2596637Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2597069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2597457Z return func(*args, **kwargs) 2025-10-10T01:01:02.2597863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2598296Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2598446Z 2025-10-10T01:01:02.2598566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2598950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2599289Z res = mod(**inputs) 2025-10-10T01:01:02.2599691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2600138Z outputs = self.model.decoder( 2025-10-10T01:01:02.2600576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2600994Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2601378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2601771Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2602184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2602582Z return func(*args, **kwargs) 2025-10-10T01:01:02.2602987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2603463Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2603657Z 2025-10-10T01:01:02.2603792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2604203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2604550Z res = mod(**inputs) 2025-10-10T01:01:02.2604942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2605368Z outputs = self.model.decoder( 2025-10-10T01:01:02.2605789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2606208Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2606583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2606979Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2607391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2607796Z return func(*args, **kwargs) 2025-10-10T01:01:02.2608208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2608672Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2609094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2609468Z return self.act(input) 2025-10-10T01:01:02.2609592Z 2025-10-10T01:01:02.2609714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2610101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2610442Z res = mod(**inputs) 2025-10-10T01:01:02.2610839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2611266Z outputs = self.model.decoder( 2025-10-10T01:01:02.2611687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2612102Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2612480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2612873Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2613279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2613680Z return func(*args, **kwargs) 2025-10-10T01:01:02.2614083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2614519Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2614677Z 2025-10-10T01:01:02.2614792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2616161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2616504Z res = mod(**inputs) 2025-10-10T01:01:02.2616909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2617339Z outputs = self.model.decoder( 2025-10-10T01:01:02.2617768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2618208Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2618593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2619008Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2619423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2619880Z return func(*args, **kwargs) 2025-10-10T01:01:02.2620304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2620765Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2621211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2621624Z return func(*args, **kwargs) 2025-10-10T01:01:02.2622048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2622574Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2622801Z 2025-10-10T01:01:02.2622920Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2623320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2623881Z res = mod(**inputs) 2025-10-10T01:01:02.2624307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2624751Z outputs = self.model.decoder( 2025-10-10T01:01:02.2625186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2625631Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2626027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2626497Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2626930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2627344Z return func(*args, **kwargs) 2025-10-10T01:01:02.2627774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2628253Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2628702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2629114Z return func(*args, **kwargs) 2025-10-10T01:01:02.2629542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2629995Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2630150Z 2025-10-10T01:01:02.2630276Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2630672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2631034Z res = mod(**inputs) 2025-10-10T01:01:02.2631555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2632088Z outputs = self.model.decoder( 2025-10-10T01:01:02.2632545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2632962Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2633344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2633738Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2634153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2634545Z return func(*args, **kwargs) 2025-10-10T01:01:02.2634959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2635411Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2635870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2636305Z return func(*args, **kwargs) 2025-10-10T01:01:02.2636712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2637178Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2637339Z 2025-10-10T01:01:02.2637430Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2637693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2638085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2638430Z res = mod(**inputs) 2025-10-10T01:01:02.2638870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2639300Z outputs = self.model.decoder( 2025-10-10T01:01:02.2639728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2640152Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2640537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2640933Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2641348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2641753Z return func(*args, **kwargs) 2025-10-10T01:01:02.2642161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2642639Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2643074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2643483Z return func(*args, **kwargs) 2025-10-10T01:01:02.2643926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2644376Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2644867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2645367Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2645555Z 2025-10-10T01:01:02.2645671Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2646040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2646379Z res = mod(**inputs) 2025-10-10T01:01:02.2646797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2647253Z outputs = self.model.decoder( 2025-10-10T01:01:02.2647681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2648080Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2648448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2648827Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2649227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2649613Z return func(*args, **kwargs) 2025-10-10T01:01:02.2650024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2650482Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2650942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2651366Z return func(*args, **kwargs) 2025-10-10T01:01:02.2651771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2652205Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2652363Z 2025-10-10T01:01:02.2652481Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2652878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2653234Z res = mod(**inputs) 2025-10-10T01:01:02.2653623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2654043Z outputs = self.model.decoder( 2025-10-10T01:01:02.2654456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2654881Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2655263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2655652Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2656065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2656462Z return func(*args, **kwargs) 2025-10-10T01:01:02.2656872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2657334Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2657527Z 2025-10-10T01:01:02.2657634Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2658007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2658359Z res = mod(**inputs) 2025-10-10T01:01:02.2658773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2659189Z outputs = self.model.decoder( 2025-10-10T01:01:02.2659635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2660085Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2660464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2660855Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2661259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2661655Z return func(*args, **kwargs) 2025-10-10T01:01:02.2662071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2662600Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2663016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2663389Z return self.act(input) 2025-10-10T01:01:02.2663518Z 2025-10-10T01:01:02.2663632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2664017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2664366Z res = mod(**inputs) 2025-10-10T01:01:02.2664775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2665215Z outputs = self.model.decoder( 2025-10-10T01:01:02.2665656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2666115Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2666572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2666975Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2667405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2667832Z return func(*args, **kwargs) 2025-10-10T01:01:02.2668250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2668676Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2668835Z 2025-10-10T01:01:02.2668949Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2669337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2669690Z res = mod(**inputs) 2025-10-10T01:01:02.2670095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2670515Z outputs = self.model.decoder( 2025-10-10T01:01:02.2670933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2671354Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2671732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2672120Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2672524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2672926Z return func(*args, **kwargs) 2025-10-10T01:01:02.2673338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2673805Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2674210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2674591Z return func(*args, **kwargs) 2025-10-10T01:01:02.2674983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2675460Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2675668Z 2025-10-10T01:01:02.2675784Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2676145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2676492Z res = mod(**inputs) 2025-10-10T01:01:02.2676892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2677374Z outputs = self.model.decoder( 2025-10-10T01:01:02.2677825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2678218Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2678596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2678989Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2679400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2679791Z return func(*args, **kwargs) 2025-10-10T01:01:02.2680185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2680621Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2681052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2681457Z return func(*args, **kwargs) 2025-10-10T01:01:02.2681841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2682254Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2682402Z 2025-10-10T01:01:02.2682511Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2682881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2683208Z res = mod(**inputs) 2025-10-10T01:01:02.2683579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2683983Z outputs = self.model.decoder( 2025-10-10T01:01:02.2684380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2684782Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2685136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2685506Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2685893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2686275Z return func(*args, **kwargs) 2025-10-10T01:01:02.2686668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2687086Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2687494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2687901Z return func(*args, **kwargs) 2025-10-10T01:01:02.2688318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2688773Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2688928Z 2025-10-10T01:01:02.2689019Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2689280Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2689660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2689984Z res = mod(**inputs) 2025-10-10T01:01:02.2690381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2690821Z outputs = self.model.decoder( 2025-10-10T01:01:02.2691297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2691698Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2692109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2692477Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2692897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2693306Z return func(*args, **kwargs) 2025-10-10T01:01:02.2693726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2694183Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2694611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2694992Z return func(*args, **kwargs) 2025-10-10T01:01:02.2695383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2695863Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2696329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2696855Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2697061Z 2025-10-10T01:01:02.2697176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2697561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2697911Z res = mod(**inputs) 2025-10-10T01:01:02.2698312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2698749Z outputs = self.model.decoder( 2025-10-10T01:01:02.2699166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2699597Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2699978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2700367Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2700779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2701189Z return func(*args, **kwargs) 2025-10-10T01:01:02.2701602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2702046Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2702482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2702889Z return func(*args, **kwargs) 2025-10-10T01:01:02.2703303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2703750Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2703904Z 2025-10-10T01:01:02.2704021Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2704421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2704781Z res = mod(**inputs) 2025-10-10T01:01:02.2705204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2705652Z outputs = self.model.decoder( 2025-10-10T01:01:02.2706091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2706614Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2707012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2707458Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2707908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2708325Z return func(*args, **kwargs) 2025-10-10T01:01:02.2708752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2709250Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2709436Z 2025-10-10T01:01:02.2709557Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2709939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2710290Z res = mod(**inputs) 2025-10-10T01:01:02.2710695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2711171Z outputs = self.model.decoder( 2025-10-10T01:01:02.2711581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2712002Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2712389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2712802Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2713213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2713619Z return func(*args, **kwargs) 2025-10-10T01:01:02.2714042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2714531Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2714971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2715360Z return self.act(input) 2025-10-10T01:01:02.2715486Z 2025-10-10T01:01:02.2715604Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2716007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2716369Z res = mod(**inputs) 2025-10-10T01:01:02.2716780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2717216Z outputs = self.model.decoder( 2025-10-10T01:01:02.2717645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2718077Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2718467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2718875Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2719294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2719707Z return func(*args, **kwargs) 2025-10-10T01:01:02.2720133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2720587Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2720740Z 2025-10-10T01:01:02.2720863Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2721253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2721611Z res = mod(**inputs) 2025-10-10T01:01:02.2722038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2722506Z outputs = self.model.decoder( 2025-10-10T01:01:02.2722950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2723387Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2723776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2724180Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2724598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2725013Z return func(*args, **kwargs) 2025-10-10T01:01:02.2725457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2725935Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2726387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2726824Z return func(*args, **kwargs) 2025-10-10T01:01:02.2727227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2727735Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2727966Z 2025-10-10T01:01:02.2728078Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2728478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2728857Z res = mod(**inputs) 2025-10-10T01:01:02.2729280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2729707Z outputs = self.model.decoder( 2025-10-10T01:01:02.2730123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2730552Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2730924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2731318Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2731936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2732349Z return func(*args, **kwargs) 2025-10-10T01:01:02.2732767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2733215Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2733650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2734053Z return func(*args, **kwargs) 2025-10-10T01:01:02.2734472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2734904Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2735060Z 2025-10-10T01:01:02.2735175Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2735565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2735917Z res = mod(**inputs) 2025-10-10T01:01:02.2736321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2736735Z outputs = self.model.decoder( 2025-10-10T01:01:02.2737150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2737578Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2737955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2738436Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2738847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2739255Z return func(*args, **kwargs) 2025-10-10T01:01:02.2739674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2740130Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2740558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2740970Z return func(*args, **kwargs) 2025-10-10T01:01:02.2741386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2741831Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2742045Z 2025-10-10T01:01:02.2742146Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2742403Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2742790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2743143Z res = mod(**inputs) 2025-10-10T01:01:02.2743557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2743999Z outputs = self.model.decoder( 2025-10-10T01:01:02.2744420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2744858Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2745252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2745660Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2746084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2746574Z return func(*args, **kwargs) 2025-10-10T01:01:02.2747004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2747476Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2747907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2748283Z return func(*args, **kwargs) 2025-10-10T01:01:02.2748675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2749109Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2749574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2750077Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2750269Z 2025-10-10T01:01:02.2750376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2750748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2751103Z res = mod(**inputs) 2025-10-10T01:01:02.2751509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2751909Z outputs = self.model.decoder( 2025-10-10T01:01:02.2752310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2752715Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2753100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2753527Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2753955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2754357Z return func(*args, **kwargs) 2025-10-10T01:01:02.2754769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2755193Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2755596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2755977Z return func(*args, **kwargs) 2025-10-10T01:01:02.2756361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2756770Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2756937Z 2025-10-10T01:01:02.2757068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2757429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2757756Z res = mod(**inputs) 2025-10-10T01:01:02.2758135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2758538Z outputs = self.model.decoder( 2025-10-10T01:01:02.2758932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2759328Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2759688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2760061Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2760448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2760834Z return func(*args, **kwargs) 2025-10-10T01:01:02.2761216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2761666Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2761847Z 2025-10-10T01:01:02.2761954Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2762388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2762715Z res = mod(**inputs) 2025-10-10T01:01:02.2763088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2763492Z outputs = self.model.decoder( 2025-10-10T01:01:02.2763876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2764281Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2764644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2765002Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2765364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2765728Z return func(*args, **kwargs) 2025-10-10T01:01:02.2766105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2766539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2766930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2767275Z return self.act(input) 2025-10-10T01:01:02.2767427Z 2025-10-10T01:01:02.2767538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2767924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2768257Z res = mod(**inputs) 2025-10-10T01:01:02.2768635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2769042Z outputs = self.model.decoder( 2025-10-10T01:01:02.2769430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2769836Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2770197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2770563Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2770972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2771384Z return func(*args, **kwargs) 2025-10-10T01:01:02.2771768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2772173Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2772315Z 2025-10-10T01:01:02.2772421Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2772797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2773125Z res = mod(**inputs) 2025-10-10T01:01:02.2773495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2773893Z outputs = self.model.decoder( 2025-10-10T01:01:02.2774284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2774695Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2775061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2775456Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2775871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2776287Z return func(*args, **kwargs) 2025-10-10T01:01:02.2776703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2777222Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2777637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2778039Z return func(*args, **kwargs) 2025-10-10T01:01:02.2778467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-10-10T01:01:02.2778997Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:02.2779218Z 2025-10-10T01:01:02.2779340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2779739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2780092Z res = mod(**inputs) 2025-10-10T01:01:02.2780505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2780947Z outputs = self.model.decoder( 2025-10-10T01:01:02.2781371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2781812Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2782205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2782656Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2783083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2783505Z return func(*args, **kwargs) 2025-10-10T01:01:02.2783939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2784411Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2784856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2785280Z return func(*args, **kwargs) 2025-10-10T01:01:02.2785719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-10-10T01:01:02.2786167Z key_states = self.k_proj(current_states) 2025-10-10T01:01:02.2786369Z 2025-10-10T01:01:02.2786582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2786997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2787379Z res = mod(**inputs) 2025-10-10T01:01:02.2787784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2788219Z outputs = self.model.decoder( 2025-10-10T01:01:02.2788665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2789099Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2789482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2789872Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2790292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2790709Z return func(*args, **kwargs) 2025-10-10T01:01:02.2791122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2791574Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2792000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2792403Z return func(*args, **kwargs) 2025-10-10T01:01:02.2792812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-10-10T01:01:02.2793260Z value_states = self.v_proj(current_states) 2025-10-10T01:01:02.2793414Z 2025-10-10T01:01:02.2793504Z cudagraph partition due to non gpu ops 2025-10-10T01:01:02.2793767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2794163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2794515Z res = mod(**inputs) 2025-10-10T01:01:02.2794915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2795339Z outputs = self.model.decoder( 2025-10-10T01:01:02.2795759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2796184Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2796564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2796952Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2797364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2797792Z return func(*args, **kwargs) 2025-10-10T01:01:02.2798238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2798703Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2799148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2799550Z return func(*args, **kwargs) 2025-10-10T01:01:02.2799961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-10-10T01:01:02.2800415Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:02.2800897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:02.2801418Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:02.2801645Z 2025-10-10T01:01:02.2801785Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2802177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2802526Z res = mod(**inputs) 2025-10-10T01:01:02.2802922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2803342Z outputs = self.model.decoder( 2025-10-10T01:01:02.2803734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2804151Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2804531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2804916Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2805331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2805741Z return func(*args, **kwargs) 2025-10-10T01:01:02.2806153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-10-10T01:01:02.2806603Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:02.2807004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2807416Z return func(*args, **kwargs) 2025-10-10T01:01:02.2807825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-10-10T01:01:02.2808273Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:02.2808427Z 2025-10-10T01:01:02.2808552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2808944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2809311Z res = mod(**inputs) 2025-10-10T01:01:02.2809745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2810174Z outputs = self.model.decoder( 2025-10-10T01:01:02.2810579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2811016Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2811394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2811785Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2812194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2812599Z return func(*args, **kwargs) 2025-10-10T01:01:02.2813015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2813543Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2813734Z 2025-10-10T01:01:02.2813855Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2814252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2814622Z res = mod(**inputs) 2025-10-10T01:01:02.2815028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2815474Z outputs = self.model.decoder( 2025-10-10T01:01:02.2815886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2816298Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2816694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2817111Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2817537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2817954Z return func(*args, **kwargs) 2025-10-10T01:01:02.2818371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-10-10T01:01:02.2818859Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:02.2819292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:02.2819681Z return self.act(input) 2025-10-10T01:01:02.2819808Z 2025-10-10T01:01:02.2819933Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2820326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2820690Z res = mod(**inputs) 2025-10-10T01:01:02.2821101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-10-10T01:01:02.2821541Z outputs = self.model.decoder( 2025-10-10T01:01:02.2821964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-10-10T01:01:02.2822402Z layer_outputs = decoder_layer( 2025-10-10T01:01:02.2822789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:02.2823198Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:02.2823620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:02.2824030Z return func(*args, **kwargs) 2025-10-10T01:01:02.2824458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-10-10T01:01:02.2824913Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:02.2825068Z 2025-10-10T01:01:02.2825193Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2825582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2825962Z res = mod(**inputs) 2025-10-10T01:01:02.2826388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1695, in forward 2025-10-10T01:01:02.2826929Z logits = self.lm_head(outputs[0]) 2025-10-10T01:01:02.2827079Z 2025-10-10T01:01:02.2827204Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:02.2827594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:02.2827953Z res = mod(**inputs) 2025-10-10T01:01:02.2828372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1701, in forward 2025-10-10T01:01:02.2828940Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T01:01:02.2829160Z 2025-10-10T01:01:12.0173607Z Compilation time (from dynamo_timed): 13.392739966 2025-10-10T01:01:12.0474743Z pass 2025-10-10T01:01:12.0475310Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:12.0476415Z TIMING: _recursive_pre_grad_passes:0.0056 _recursive_joint_graph_passes:0.2718 _recursive_post_grad_passes:0.05121 async_compile.wait:0.78329 code_gen:9.13989 inductor_compile:10.19944 backend_compile:12.08536 gc:0.0013 entire_frame_compile:13.39274 total_wall_time:13.39274 2025-10-10T01:01:12.0477487Z STATS: call_* op count: 180 | FakeTensorMode.__torch_dispatch__:4126 | FakeTensor.__torch_dispatch__:2342 | ProxyTorchDispatchMode.__torch_dispatch__:1073 2025-10-10T01:01:12.0478073Z Dynamo produced 1 graphs covering 180 ops with 0 graph breaks (0 unique) 2025-10-10T01:01:14.7556236Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:01:14.7557136Z import pynvml # type: ignore[import] 2025-10-10T01:01:18.3274346Z 2025-10-10T01:01:21.9256144Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:01:21.9256477Z loading model: 0it [00:03, ?it/s] 2025-10-10T01:01:21.9261218Z cpu eval PegasusForCausalLM 2025-10-10T01:01:22.2863689Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:22.4646994Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:22.6036377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:30.4953078Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4953418Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4953659Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4953895Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4954125Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4954348Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4954580Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4954812Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4955041Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4955760Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4956016Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4956242Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4956510Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4956932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4957313Z res = mod(**inputs) 2025-10-10T01:01:30.4957769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4958227Z outputs = self.model.decoder( 2025-10-10T01:01:30.4958672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4959110Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4959498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4959908Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4960337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4960752Z return func(*args, **kwargs) 2025-10-10T01:01:30.4961177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.4962044Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.4962492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4962917Z return func(*args, **kwargs) 2025-10-10T01:01:30.4963356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.4963877Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.4964114Z 2025-10-10T01:01:30.4964236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4964633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4964988Z res = mod(**inputs) 2025-10-10T01:01:30.4965452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4965941Z outputs = self.model.decoder( 2025-10-10T01:01:30.4966368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4966797Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4967186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4967588Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4968008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4968416Z return func(*args, **kwargs) 2025-10-10T01:01:30.4968836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.4969318Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.4969763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4970190Z return func(*args, **kwargs) 2025-10-10T01:01:30.4970623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.4971093Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.4971242Z 2025-10-10T01:01:30.4971365Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4971752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4972115Z res = mod(**inputs) 2025-10-10T01:01:30.4972523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4972963Z outputs = self.model.decoder( 2025-10-10T01:01:30.4973406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4973842Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4974225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4974640Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4975063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4975482Z return func(*args, **kwargs) 2025-10-10T01:01:30.4975925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.4976411Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.4976843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4977474Z return func(*args, **kwargs) 2025-10-10T01:01:30.4977923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.4978386Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.4978553Z 2025-10-10T01:01:30.4978646Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.4978917Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4979321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4979682Z res = mod(**inputs) 2025-10-10T01:01:30.4980114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4980562Z outputs = self.model.decoder( 2025-10-10T01:01:30.4981001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4981520Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4981911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4982316Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4982741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4983158Z return func(*args, **kwargs) 2025-10-10T01:01:30.4983577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.4984046Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.4984487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4984912Z return func(*args, **kwargs) 2025-10-10T01:01:30.4985343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.4985978Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.4986775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.4987346Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.4987553Z 2025-10-10T01:01:30.4987682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4988084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4988440Z res = mod(**inputs) 2025-10-10T01:01:30.4988864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4989315Z outputs = self.model.decoder( 2025-10-10T01:01:30.4989761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4990197Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4990591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4990994Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4991422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4991838Z return func(*args, **kwargs) 2025-10-10T01:01:30.4992260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.4992728Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.4993171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4993627Z return func(*args, **kwargs) 2025-10-10T01:01:30.4994080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.4994531Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.4994695Z 2025-10-10T01:01:30.4994813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.4995213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.4995568Z res = mod(**inputs) 2025-10-10T01:01:30.4995978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.4996432Z outputs = self.model.decoder( 2025-10-10T01:01:30.4996871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.4997315Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.4997762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.4998162Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.4998567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.4998989Z return func(*args, **kwargs) 2025-10-10T01:01:30.4999399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.4999868Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5000062Z 2025-10-10T01:01:30.5000176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5000562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5000907Z res = mod(**inputs) 2025-10-10T01:01:30.5001318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5001765Z outputs = self.model.decoder( 2025-10-10T01:01:30.5002210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5002660Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5003046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5003440Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5003860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5004278Z return func(*args, **kwargs) 2025-10-10T01:01:30.5004703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5005192Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5005604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5005974Z return self.act(input) 2025-10-10T01:01:30.5006100Z 2025-10-10T01:01:30.5006212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5006595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5006933Z res = mod(**inputs) 2025-10-10T01:01:30.5007343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5007780Z outputs = self.model.decoder( 2025-10-10T01:01:30.5008210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5008652Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5009093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5009498Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5009917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5010327Z return func(*args, **kwargs) 2025-10-10T01:01:30.5010753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5011187Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5011348Z 2025-10-10T01:01:30.5011464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5011860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5012220Z res = mod(**inputs) 2025-10-10T01:01:30.5013288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5013788Z outputs = self.model.decoder( 2025-10-10T01:01:30.5014226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5014672Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5015066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5015472Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5015900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5016320Z return func(*args, **kwargs) 2025-10-10T01:01:30.5016751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5017229Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5017671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5018085Z return func(*args, **kwargs) 2025-10-10T01:01:30.5018515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5019053Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5019280Z 2025-10-10T01:01:30.5019399Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5019803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5020166Z res = mod(**inputs) 2025-10-10T01:01:30.5020586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5021033Z outputs = self.model.decoder( 2025-10-10T01:01:30.5021470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5021910Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5022302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5022715Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5023143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5023562Z return func(*args, **kwargs) 2025-10-10T01:01:30.5023991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5024465Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5024911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5025371Z return func(*args, **kwargs) 2025-10-10T01:01:30.5025804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5026255Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5026563Z 2025-10-10T01:01:30.5026697Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5027095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5027448Z res = mod(**inputs) 2025-10-10T01:01:30.5027865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5028308Z outputs = self.model.decoder( 2025-10-10T01:01:30.5028742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5029239Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5029627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5030029Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5030460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5030872Z return func(*args, **kwargs) 2025-10-10T01:01:30.5031291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5031982Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5032433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5032869Z return func(*args, **kwargs) 2025-10-10T01:01:30.5033289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5033733Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5033898Z 2025-10-10T01:01:30.5033989Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5034254Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5034643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5034984Z res = mod(**inputs) 2025-10-10T01:01:30.5035391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5035824Z outputs = self.model.decoder( 2025-10-10T01:01:30.5036248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5036674Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5037050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5037448Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5037858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5038258Z return func(*args, **kwargs) 2025-10-10T01:01:30.5038674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5039128Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5039561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5039975Z return func(*args, **kwargs) 2025-10-10T01:01:30.5040388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5040913Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5041430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5041969Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5042178Z 2025-10-10T01:01:30.5042295Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5042688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5043034Z res = mod(**inputs) 2025-10-10T01:01:30.5043445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5043889Z outputs = self.model.decoder( 2025-10-10T01:01:30.5044320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5044816Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5045194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5045590Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5046005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5046412Z return func(*args, **kwargs) 2025-10-10T01:01:30.5046842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5047310Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5047744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5048148Z return func(*args, **kwargs) 2025-10-10T01:01:30.5048567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5049007Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5049168Z 2025-10-10T01:01:30.5049282Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5049670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5050019Z res = mod(**inputs) 2025-10-10T01:01:30.5050425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5050852Z outputs = self.model.decoder( 2025-10-10T01:01:30.5051276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5051710Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5052094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5052497Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5052908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5053317Z return func(*args, **kwargs) 2025-10-10T01:01:30.5053737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5054217Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5054406Z 2025-10-10T01:01:30.5054520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5054909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5055260Z res = mod(**inputs) 2025-10-10T01:01:30.5055667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5056126Z outputs = self.model.decoder( 2025-10-10T01:01:30.5056560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5056991Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5057369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5057762Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5058177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5058600Z return func(*args, **kwargs) 2025-10-10T01:01:30.5059012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5059489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5059932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5060347Z return self.act(input) 2025-10-10T01:01:30.5060481Z 2025-10-10T01:01:30.5060598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5060995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5061354Z res = mod(**inputs) 2025-10-10T01:01:30.5061769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5062214Z outputs = self.model.decoder( 2025-10-10T01:01:30.5062660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5063101Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5063488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5063892Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5064318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5064735Z return func(*args, **kwargs) 2025-10-10T01:01:30.5065158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5065619Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5065772Z 2025-10-10T01:01:30.5065888Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5066284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5066748Z res = mod(**inputs) 2025-10-10T01:01:30.5067175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5067627Z outputs = self.model.decoder( 2025-10-10T01:01:30.5068060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5068516Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5068899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5069300Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5069728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5070198Z return func(*args, **kwargs) 2025-10-10T01:01:30.5070639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5071110Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5071567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5072024Z return func(*args, **kwargs) 2025-10-10T01:01:30.5072453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5072983Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5073230Z 2025-10-10T01:01:30.5073384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5073792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5074155Z res = mod(**inputs) 2025-10-10T01:01:30.5074570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5075024Z outputs = self.model.decoder( 2025-10-10T01:01:30.5075483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5075953Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5076369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5076772Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5077209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5077634Z return func(*args, **kwargs) 2025-10-10T01:01:30.5078065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5078576Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5079031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5079458Z return func(*args, **kwargs) 2025-10-10T01:01:30.5079890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5080335Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5080496Z 2025-10-10T01:01:30.5080613Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5081009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5081389Z res = mod(**inputs) 2025-10-10T01:01:30.5081797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5082240Z outputs = self.model.decoder( 2025-10-10T01:01:30.5082674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5083114Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5083500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5083901Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5084324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5084740Z return func(*args, **kwargs) 2025-10-10T01:01:30.5085160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5085626Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5086072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5086481Z return func(*args, **kwargs) 2025-10-10T01:01:30.5086910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5087374Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5087539Z 2025-10-10T01:01:30.5087658Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5087918Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5088314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5088672Z res = mod(**inputs) 2025-10-10T01:01:30.5089092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5089526Z outputs = self.model.decoder( 2025-10-10T01:01:30.5089960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5090397Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5090787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5091223Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5091651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5092056Z return func(*args, **kwargs) 2025-10-10T01:01:30.5092479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5092947Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5093385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5093789Z return func(*args, **kwargs) 2025-10-10T01:01:30.5094199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5094661Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5095159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5095695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5095910Z 2025-10-10T01:01:30.5096029Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5096431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5096788Z res = mod(**inputs) 2025-10-10T01:01:30.5097207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5097646Z outputs = self.model.decoder( 2025-10-10T01:01:30.5098088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5098526Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5098918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5099322Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5099751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5100160Z return func(*args, **kwargs) 2025-10-10T01:01:30.5100579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5101037Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5101462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5101876Z return func(*args, **kwargs) 2025-10-10T01:01:30.5102320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5102792Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5102953Z 2025-10-10T01:01:30.5103096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5103487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5103848Z res = mod(**inputs) 2025-10-10T01:01:30.5104260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5104705Z outputs = self.model.decoder( 2025-10-10T01:01:30.5105133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5105572Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5105961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5106464Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5106967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5107395Z return func(*args, **kwargs) 2025-10-10T01:01:30.5107827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5108329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5108524Z 2025-10-10T01:01:30.5108650Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5109051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5109424Z res = mod(**inputs) 2025-10-10T01:01:30.5109840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5110284Z outputs = self.model.decoder( 2025-10-10T01:01:30.5110724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5111165Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5111561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5111970Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5112395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5112807Z return func(*args, **kwargs) 2025-10-10T01:01:30.5113229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5113720Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5114155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5114543Z return self.act(input) 2025-10-10T01:01:30.5114669Z 2025-10-10T01:01:30.5114796Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5115189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5115550Z res = mod(**inputs) 2025-10-10T01:01:30.5115966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5116411Z outputs = self.model.decoder( 2025-10-10T01:01:30.5116836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5117279Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5117657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5118052Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5118467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5118915Z return func(*args, **kwargs) 2025-10-10T01:01:30.5119333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5119769Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5119921Z 2025-10-10T01:01:30.5120042Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5120429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5120771Z res = mod(**inputs) 2025-10-10T01:01:30.5121176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5121604Z outputs = self.model.decoder( 2025-10-10T01:01:30.5122060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5122496Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5122885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5123290Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5123717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5124120Z return func(*args, **kwargs) 2025-10-10T01:01:30.5124545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-10-10T01:01:30.5124983Z hidden_states = residual + hidden_states 2025-10-10T01:01:30.5125140Z 2025-10-10T01:01:30.5125253Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5125636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5125981Z res = mod(**inputs) 2025-10-10T01:01:30.5126387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5126813Z outputs = self.model.decoder( 2025-10-10T01:01:30.5127232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5127655Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5128026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5128425Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5128836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5129292Z return func(*args, **kwargs) 2025-10-10T01:01:30.5129725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5130181Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5130609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5131022Z return func(*args, **kwargs) 2025-10-10T01:01:30.5131437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5132102Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5132331Z 2025-10-10T01:01:30.5132448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5132850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5133219Z res = mod(**inputs) 2025-10-10T01:01:30.5133638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5134171Z outputs = self.model.decoder( 2025-10-10T01:01:30.5134601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5135031Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5135414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5135809Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5136220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5136628Z return func(*args, **kwargs) 2025-10-10T01:01:30.5137046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5137504Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5137997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5138395Z return func(*args, **kwargs) 2025-10-10T01:01:30.5138810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5139246Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5139393Z 2025-10-10T01:01:30.5139514Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5139893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5140245Z res = mod(**inputs) 2025-10-10T01:01:30.5140652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5141083Z outputs = self.model.decoder( 2025-10-10T01:01:30.5141512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5141939Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5142318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5142721Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5143141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5143557Z return func(*args, **kwargs) 2025-10-10T01:01:30.5144000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5144471Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5144923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5145339Z return func(*args, **kwargs) 2025-10-10T01:01:30.5145777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5146237Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5146464Z 2025-10-10T01:01:30.5146567Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5146846Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5147258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5147622Z res = mod(**inputs) 2025-10-10T01:01:30.5148040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5148489Z outputs = self.model.decoder( 2025-10-10T01:01:30.5148932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5149409Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5149821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5150226Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5150650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5151101Z return func(*args, **kwargs) 2025-10-10T01:01:30.5151556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5152027Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5152493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5152941Z return func(*args, **kwargs) 2025-10-10T01:01:30.5153391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5153856Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5154314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5154814Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5155002Z 2025-10-10T01:01:30.5155120Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5155486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5155816Z res = mod(**inputs) 2025-10-10T01:01:30.5156201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5156611Z outputs = self.model.decoder( 2025-10-10T01:01:30.5157017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5157420Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5157781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5158155Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5158542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5158924Z return func(*args, **kwargs) 2025-10-10T01:01:30.5159310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5159749Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5160188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5160601Z return func(*args, **kwargs) 2025-10-10T01:01:30.5161011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5161454Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5161619Z 2025-10-10T01:01:30.5161727Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5162098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5162430Z res = mod(**inputs) 2025-10-10T01:01:30.5162811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5163247Z outputs = self.model.decoder( 2025-10-10T01:01:30.5163673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5164106Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5164531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5164926Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5165318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5165716Z return func(*args, **kwargs) 2025-10-10T01:01:30.5166141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5166612Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5166807Z 2025-10-10T01:01:30.5166922Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5167309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5167656Z res = mod(**inputs) 2025-10-10T01:01:30.5168081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5168529Z outputs = self.model.decoder( 2025-10-10T01:01:30.5168963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5169399Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5169783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5170175Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5170567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5170978Z return func(*args, **kwargs) 2025-10-10T01:01:30.5171397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5171885Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5172313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5172694Z return self.act(input) 2025-10-10T01:01:30.5172826Z 2025-10-10T01:01:30.5172944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5173339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5173696Z res = mod(**inputs) 2025-10-10T01:01:30.5174102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5174540Z outputs = self.model.decoder( 2025-10-10T01:01:30.5174969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5175402Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5175796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5176191Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5176610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5177025Z return func(*args, **kwargs) 2025-10-10T01:01:30.5177453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5177892Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5178054Z 2025-10-10T01:01:30.5178172Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5178574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5178935Z res = mod(**inputs) 2025-10-10T01:01:30.5179349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5179834Z outputs = self.model.decoder( 2025-10-10T01:01:30.5180256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5180683Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5181064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5181459Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5181871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5182273Z return func(*args, **kwargs) 2025-10-10T01:01:30.5182690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5183155Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5183661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5184085Z return func(*args, **kwargs) 2025-10-10T01:01:30.5184519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5185031Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5185252Z 2025-10-10T01:01:30.5185373Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5185752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5186100Z res = mod(**inputs) 2025-10-10T01:01:30.5186592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5187053Z outputs = self.model.decoder( 2025-10-10T01:01:30.5187500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5187945Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5188325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5188717Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5189133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5189536Z return func(*args, **kwargs) 2025-10-10T01:01:30.5189958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5190413Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5190850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5191258Z return func(*args, **kwargs) 2025-10-10T01:01:30.5191666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5192103Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5192257Z 2025-10-10T01:01:30.5192371Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5192756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5193103Z res = mod(**inputs) 2025-10-10T01:01:30.5193499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5193927Z outputs = self.model.decoder( 2025-10-10T01:01:30.5194351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5194826Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5195223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5195617Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5196027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5196435Z return func(*args, **kwargs) 2025-10-10T01:01:30.5196847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5197293Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5197729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5198144Z return func(*args, **kwargs) 2025-10-10T01:01:30.5198598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5199083Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5199243Z 2025-10-10T01:01:30.5199336Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5199603Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5200010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5200357Z res = mod(**inputs) 2025-10-10T01:01:30.5200759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5201204Z outputs = self.model.decoder( 2025-10-10T01:01:30.5201638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5202079Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5202470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5202869Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5203299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5203713Z return func(*args, **kwargs) 2025-10-10T01:01:30.5204138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5204605Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5205045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5205463Z return func(*args, **kwargs) 2025-10-10T01:01:30.5205889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5206362Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5206854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5207394Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5207607Z 2025-10-10T01:01:30.5207724Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5208124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5208479Z res = mod(**inputs) 2025-10-10T01:01:30.5208887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5209332Z outputs = self.model.decoder( 2025-10-10T01:01:30.5209767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5210238Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5210645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5211050Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5211480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5211910Z return func(*args, **kwargs) 2025-10-10T01:01:30.5212342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5212809Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5213264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5213684Z return func(*args, **kwargs) 2025-10-10T01:01:30.5214133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5214608Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5214760Z 2025-10-10T01:01:30.5214875Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5215263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5215615Z res = mod(**inputs) 2025-10-10T01:01:30.5216034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5216478Z outputs = self.model.decoder( 2025-10-10T01:01:30.5216909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5217349Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5217739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5218156Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5218585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5219008Z return func(*args, **kwargs) 2025-10-10T01:01:30.5219444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5219947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5220141Z 2025-10-10T01:01:30.5220266Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5220674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5221049Z res = mod(**inputs) 2025-10-10T01:01:30.5221467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5221928Z outputs = self.model.decoder( 2025-10-10T01:01:30.5222369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5222823Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5223214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5223628Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5224058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5224468Z return func(*args, **kwargs) 2025-10-10T01:01:30.5224909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5225407Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5225846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5226286Z return self.act(input) 2025-10-10T01:01:30.5226488Z 2025-10-10T01:01:30.5226611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5227019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5227394Z res = mod(**inputs) 2025-10-10T01:01:30.5227829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5228267Z outputs = self.model.decoder( 2025-10-10T01:01:30.5228683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5229122Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5229515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5229972Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5230383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5230789Z return func(*args, **kwargs) 2025-10-10T01:01:30.5231214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5231875Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5232034Z 2025-10-10T01:01:30.5232162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5232554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5232917Z res = mod(**inputs) 2025-10-10T01:01:30.5233336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5233788Z outputs = self.model.decoder( 2025-10-10T01:01:30.5234231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5234677Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5235072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5235476Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5235899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5236318Z return func(*args, **kwargs) 2025-10-10T01:01:30.5236748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-10-10T01:01:30.5237203Z hidden_states = residual + hidden_states 2025-10-10T01:01:30.5237360Z 2025-10-10T01:01:30.5237485Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5237891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5238244Z res = mod(**inputs) 2025-10-10T01:01:30.5238658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5239096Z outputs = self.model.decoder( 2025-10-10T01:01:30.5239533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5239970Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5240360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5240765Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5241193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5241681Z return func(*args, **kwargs) 2025-10-10T01:01:30.5242134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5242610Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5243061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5243480Z return func(*args, **kwargs) 2025-10-10T01:01:30.5243911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5244432Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5244668Z 2025-10-10T01:01:30.5244786Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5245188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5245553Z res = mod(**inputs) 2025-10-10T01:01:30.5246038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5246476Z outputs = self.model.decoder( 2025-10-10T01:01:30.5246900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5247332Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5247714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5248101Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5248519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5248938Z return func(*args, **kwargs) 2025-10-10T01:01:30.5249379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5249845Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5250276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5250688Z return func(*args, **kwargs) 2025-10-10T01:01:30.5251105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5251546Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5251694Z 2025-10-10T01:01:30.5251815Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5252207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5252557Z res = mod(**inputs) 2025-10-10T01:01:30.5252964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5253406Z outputs = self.model.decoder( 2025-10-10T01:01:30.5253832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5254266Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5254645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5255047Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5255465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5255866Z return func(*args, **kwargs) 2025-10-10T01:01:30.5256287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5256748Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5257222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5257647Z return func(*args, **kwargs) 2025-10-10T01:01:30.5258080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5258530Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5258684Z 2025-10-10T01:01:30.5258779Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5259041Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5259423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5259772Z res = mod(**inputs) 2025-10-10T01:01:30.5260180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5260611Z outputs = self.model.decoder( 2025-10-10T01:01:30.5261061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5261501Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5261889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5262297Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5262733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5263155Z return func(*args, **kwargs) 2025-10-10T01:01:30.5263599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5264075Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5264534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5264965Z return func(*args, **kwargs) 2025-10-10T01:01:30.5265388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5265873Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5266548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5267111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5267318Z 2025-10-10T01:01:30.5267448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5267839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5268205Z res = mod(**inputs) 2025-10-10T01:01:30.5268628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5269088Z outputs = self.model.decoder( 2025-10-10T01:01:30.5269522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5269966Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5270358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5270769Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5271196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5271629Z return func(*args, **kwargs) 2025-10-10T01:01:30.5272059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5272534Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5272982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5273463Z return func(*args, **kwargs) 2025-10-10T01:01:30.5273884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5274334Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5274496Z 2025-10-10T01:01:30.5274612Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5275008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5275357Z res = mod(**inputs) 2025-10-10T01:01:30.5275772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5276213Z outputs = self.model.decoder( 2025-10-10T01:01:30.5276666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5277118Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5277498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5277898Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5278316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5278737Z return func(*args, **kwargs) 2025-10-10T01:01:30.5279173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5279655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5279854Z 2025-10-10T01:01:30.5279971Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5280369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5280729Z res = mod(**inputs) 2025-10-10T01:01:30.5281136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5281575Z outputs = self.model.decoder( 2025-10-10T01:01:30.5282007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5282441Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5282825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5283222Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5283639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5284053Z return func(*args, **kwargs) 2025-10-10T01:01:30.5284479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5284967Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5285390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5285772Z return self.act(input) 2025-10-10T01:01:30.5285903Z 2025-10-10T01:01:30.5286021Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5286420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5286768Z res = mod(**inputs) 2025-10-10T01:01:30.5287180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5287619Z outputs = self.model.decoder( 2025-10-10T01:01:30.5288049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5288509Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5288902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5289296Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5289706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5290109Z return func(*args, **kwargs) 2025-10-10T01:01:30.5290525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5290955Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5291111Z 2025-10-10T01:01:30.5291224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5291610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5291995Z res = mod(**inputs) 2025-10-10T01:01:30.5292416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5292849Z outputs = self.model.decoder( 2025-10-10T01:01:30.5293271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5293696Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5294071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5294463Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5294874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5295283Z return func(*args, **kwargs) 2025-10-10T01:01:30.5295706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5296178Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5296617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5297028Z return func(*args, **kwargs) 2025-10-10T01:01:30.5297463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5297986Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5298214Z 2025-10-10T01:01:30.5298332Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5298731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5299091Z res = mod(**inputs) 2025-10-10T01:01:30.5299512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5299959Z outputs = self.model.decoder( 2025-10-10T01:01:30.5300388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5300829Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5301217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5301626Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5302058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5302472Z return func(*args, **kwargs) 2025-10-10T01:01:30.5302910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5303374Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5303862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5304287Z return func(*args, **kwargs) 2025-10-10T01:01:30.5304735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5305211Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5305366Z 2025-10-10T01:01:30.5305494Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5305889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5306256Z res = mod(**inputs) 2025-10-10T01:01:30.5306785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5307256Z outputs = self.model.decoder( 2025-10-10T01:01:30.5307795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5308275Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5308672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5309093Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5309532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5309963Z return func(*args, **kwargs) 2025-10-10T01:01:30.5310393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5310873Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5311315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5311729Z return func(*args, **kwargs) 2025-10-10T01:01:30.5312158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5312607Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5312776Z 2025-10-10T01:01:30.5312868Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5313135Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5313534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5313903Z res = mod(**inputs) 2025-10-10T01:01:30.5314315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5314759Z outputs = self.model.decoder( 2025-10-10T01:01:30.5315191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5315637Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5316028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5316418Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5316829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5317229Z return func(*args, **kwargs) 2025-10-10T01:01:30.5317642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5318096Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5318551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5318962Z return func(*args, **kwargs) 2025-10-10T01:01:30.5319378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5319882Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5320369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5320894Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5321099Z 2025-10-10T01:01:30.5321213Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5321604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5321954Z res = mod(**inputs) 2025-10-10T01:01:30.5322375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5322805Z outputs = self.model.decoder( 2025-10-10T01:01:30.5323283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5323739Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5324120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5324522Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5324944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5325354Z return func(*args, **kwargs) 2025-10-10T01:01:30.5325784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5326263Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5326719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5327146Z return func(*args, **kwargs) 2025-10-10T01:01:30.5327589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5328043Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5328210Z 2025-10-10T01:01:30.5328331Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5328736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5329107Z res = mod(**inputs) 2025-10-10T01:01:30.5329536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5329992Z outputs = self.model.decoder( 2025-10-10T01:01:30.5330424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5330858Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5331253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5331862Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5332277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5332685Z return func(*args, **kwargs) 2025-10-10T01:01:30.5333121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5333617Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5333805Z 2025-10-10T01:01:30.5333921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5334311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5334665Z res = mod(**inputs) 2025-10-10T01:01:30.5335078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5335586Z outputs = self.model.decoder( 2025-10-10T01:01:30.5336004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5336436Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5336814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5337206Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5337613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5338018Z return func(*args, **kwargs) 2025-10-10T01:01:30.5338436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5338939Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5339396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5339762Z return self.act(input) 2025-10-10T01:01:30.5339890Z 2025-10-10T01:01:30.5340004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5340396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5340749Z res = mod(**inputs) 2025-10-10T01:01:30.5341158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5341587Z outputs = self.model.decoder( 2025-10-10T01:01:30.5342008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5342439Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5342835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5343237Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5343664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5344080Z return func(*args, **kwargs) 2025-10-10T01:01:30.5344521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5344971Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5345126Z 2025-10-10T01:01:30.5345245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5345649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5346008Z res = mod(**inputs) 2025-10-10T01:01:30.5346490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5346950Z outputs = self.model.decoder( 2025-10-10T01:01:30.5347379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5347911Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5348307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5348715Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5349133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5349551Z return func(*args, **kwargs) 2025-10-10T01:01:30.5349985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-10-10T01:01:30.5350437Z hidden_states = residual + hidden_states 2025-10-10T01:01:30.5350618Z 2025-10-10T01:01:30.5350743Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5351152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5351510Z res = mod(**inputs) 2025-10-10T01:01:30.5351922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5352363Z outputs = self.model.decoder( 2025-10-10T01:01:30.5352790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5353221Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5353612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5354012Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5354474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5354905Z return func(*args, **kwargs) 2025-10-10T01:01:30.5355332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5355801Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5356252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5356673Z return func(*args, **kwargs) 2025-10-10T01:01:30.5357092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5357621Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5357850Z 2025-10-10T01:01:30.5357965Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5358359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5358715Z res = mod(**inputs) 2025-10-10T01:01:30.5359116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5359548Z outputs = self.model.decoder( 2025-10-10T01:01:30.5359971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5360400Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5360769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5361170Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5361581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5361983Z return func(*args, **kwargs) 2025-10-10T01:01:30.5362405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5362851Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5363284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5363688Z return func(*args, **kwargs) 2025-10-10T01:01:30.5364105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5364541Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5364686Z 2025-10-10T01:01:30.5364799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5365184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5365534Z res = mod(**inputs) 2025-10-10T01:01:30.5365975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5366405Z outputs = self.model.decoder( 2025-10-10T01:01:30.5366832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5367261Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5367640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5368031Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5368438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5368840Z return func(*args, **kwargs) 2025-10-10T01:01:30.5369256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5369748Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5370183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5370573Z return func(*args, **kwargs) 2025-10-10T01:01:30.5370981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5371417Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5371570Z 2025-10-10T01:01:30.5371665Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5371917Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5372303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5372652Z res = mod(**inputs) 2025-10-10T01:01:30.5373054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5373488Z outputs = self.model.decoder( 2025-10-10T01:01:30.5373901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5374327Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5374703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5375102Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5375506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5375907Z return func(*args, **kwargs) 2025-10-10T01:01:30.5376322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5376774Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5377210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5377607Z return func(*args, **kwargs) 2025-10-10T01:01:30.5378022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5378478Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5378958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5379485Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5379490Z 2025-10-10T01:01:30.5379601Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5379816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5379906Z res = mod(**inputs) 2025-10-10T01:01:30.5380232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5380327Z outputs = self.model.decoder( 2025-10-10T01:01:30.5380615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5380701Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5380941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5381030Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5381297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5381372Z return func(*args, **kwargs) 2025-10-10T01:01:30.5381658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5381803Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5382064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5382147Z return func(*args, **kwargs) 2025-10-10T01:01:30.5382430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5382528Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5382532Z 2025-10-10T01:01:30.5382646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5382858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5382937Z res = mod(**inputs) 2025-10-10T01:01:30.5383239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5383326Z outputs = self.model.decoder( 2025-10-10T01:01:30.5383617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5383703Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5383945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5384032Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5384301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5384376Z return func(*args, **kwargs) 2025-10-10T01:01:30.5384666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5384796Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5384800Z 2025-10-10T01:01:30.5384912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5385139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5385210Z res = mod(**inputs) 2025-10-10T01:01:30.5385504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5385582Z outputs = self.model.decoder( 2025-10-10T01:01:30.5385879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5385959Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5386206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5386302Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5386651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5386769Z return func(*args, **kwargs) 2025-10-10T01:01:30.5387083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5387221Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5387470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5387550Z return self.act(input) 2025-10-10T01:01:30.5387554Z 2025-10-10T01:01:30.5387837Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5388062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5388141Z res = mod(**inputs) 2025-10-10T01:01:30.5388431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5388511Z outputs = self.model.decoder( 2025-10-10T01:01:30.5388842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5388922Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5389168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5389256Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5389518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5389602Z return func(*args, **kwargs) 2025-10-10T01:01:30.5389886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5389981Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5389985Z 2025-10-10T01:01:30.5390097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5390320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5390394Z res = mod(**inputs) 2025-10-10T01:01:30.5390681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5390767Z outputs = self.model.decoder( 2025-10-10T01:01:30.5391054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5391139Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5391378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5391464Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5391735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5391810Z return func(*args, **kwargs) 2025-10-10T01:01:30.5392106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5392215Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5392476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5392559Z return func(*args, **kwargs) 2025-10-10T01:01:30.5392839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5393012Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5393016Z 2025-10-10T01:01:30.5393128Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5393347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5393440Z res = mod(**inputs) 2025-10-10T01:01:30.5393769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5393862Z outputs = self.model.decoder( 2025-10-10T01:01:30.5394154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5394241Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5394485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5394574Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5394851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5394928Z return func(*args, **kwargs) 2025-10-10T01:01:30.5395222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5395370Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5395651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5395726Z return func(*args, **kwargs) 2025-10-10T01:01:30.5396002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5396096Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5396099Z 2025-10-10T01:01:30.5396209Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5396426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5396496Z res = mod(**inputs) 2025-10-10T01:01:30.5396777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5396868Z outputs = self.model.decoder( 2025-10-10T01:01:30.5397150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5397236Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5397471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5397557Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5397821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5397896Z return func(*args, **kwargs) 2025-10-10T01:01:30.5398184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5398288Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5398555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5398634Z return func(*args, **kwargs) 2025-10-10T01:01:30.5398916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5399018Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5399022Z 2025-10-10T01:01:30.5399113Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5399232Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5399451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5399522Z res = mod(**inputs) 2025-10-10T01:01:30.5399822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5399904Z outputs = self.model.decoder( 2025-10-10T01:01:30.5400213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5400331Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5400578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5400672Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5400941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5401026Z return func(*args, **kwargs) 2025-10-10T01:01:30.5401316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5401432Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5401699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5401795Z return func(*args, **kwargs) 2025-10-10T01:01:30.5402109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5402222Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5402555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5402705Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5402709Z 2025-10-10T01:01:30.5402830Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5403047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5403119Z res = mod(**inputs) 2025-10-10T01:01:30.5403420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5403506Z outputs = self.model.decoder( 2025-10-10T01:01:30.5403806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5403885Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5404128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5404226Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5404492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5404576Z return func(*args, **kwargs) 2025-10-10T01:01:30.5404863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5404971Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5405246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5405328Z return func(*args, **kwargs) 2025-10-10T01:01:30.5405625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5405717Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5405721Z 2025-10-10T01:01:30.5405843Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5406064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5406136Z res = mod(**inputs) 2025-10-10T01:01:30.5406437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5406519Z outputs = self.model.decoder( 2025-10-10T01:01:30.5406818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5406923Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5407187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5407283Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5407552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5407637Z return func(*args, **kwargs) 2025-10-10T01:01:30.5407926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5408058Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5408069Z 2025-10-10T01:01:30.5408184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5408412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5408537Z res = mod(**inputs) 2025-10-10T01:01:30.5408832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5408921Z outputs = self.model.decoder( 2025-10-10T01:01:30.5409213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5409295Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5409549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5409638Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5409913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5409992Z return func(*args, **kwargs) 2025-10-10T01:01:30.5410281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5410428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5410665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5410752Z return self.act(input) 2025-10-10T01:01:30.5410755Z 2025-10-10T01:01:30.5410869Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5411093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5411165Z res = mod(**inputs) 2025-10-10T01:01:30.5411459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5411547Z outputs = self.model.decoder( 2025-10-10T01:01:30.5411836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5411928Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5412172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5412261Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5412544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5412622Z return func(*args, **kwargs) 2025-10-10T01:01:30.5412916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5413006Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5413010Z 2025-10-10T01:01:30.5413124Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5413356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5413450Z res = mod(**inputs) 2025-10-10T01:01:30.5413768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5413851Z outputs = self.model.decoder( 2025-10-10T01:01:30.5414149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5414229Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5414473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5414569Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5414834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5414921Z return func(*args, **kwargs) 2025-10-10T01:01:30.5415209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-10-10T01:01:30.5415339Z hidden_states = residual + hidden_states 2025-10-10T01:01:30.5415345Z 2025-10-10T01:01:30.5415471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5415688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5415767Z res = mod(**inputs) 2025-10-10T01:01:30.5416058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5416146Z outputs = self.model.decoder( 2025-10-10T01:01:30.5416440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5416519Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5416772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5416863Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5417140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5417216Z return func(*args, **kwargs) 2025-10-10T01:01:30.5417507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5417625Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5417894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5417978Z return func(*args, **kwargs) 2025-10-10T01:01:30.5418268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5418436Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5418449Z 2025-10-10T01:01:30.5418568Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5418788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5418867Z res = mod(**inputs) 2025-10-10T01:01:30.5419161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5419246Z outputs = self.model.decoder( 2025-10-10T01:01:30.5419550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5419631Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5419882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5419970Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5420246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5420369Z return func(*args, **kwargs) 2025-10-10T01:01:30.5420661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5420777Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5421043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5421126Z return func(*args, **kwargs) 2025-10-10T01:01:30.5421413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5421510Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5421514Z 2025-10-10T01:01:30.5421628Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5421845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5421964Z res = mod(**inputs) 2025-10-10T01:01:30.5422262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5422351Z outputs = self.model.decoder( 2025-10-10T01:01:30.5422643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5422724Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5422974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5423060Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5423335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5423412Z return func(*args, **kwargs) 2025-10-10T01:01:30.5423700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5423820Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5424085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5424169Z return func(*args, **kwargs) 2025-10-10T01:01:30.5424456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5424559Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5424563Z 2025-10-10T01:01:30.5424654Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5424770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5424996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5425069Z res = mod(**inputs) 2025-10-10T01:01:30.5425373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5425455Z outputs = self.model.decoder( 2025-10-10T01:01:30.5425748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5425838Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5426080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5426175Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5426524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5426616Z return func(*args, **kwargs) 2025-10-10T01:01:30.5426906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5427043Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5427338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5427418Z return func(*args, **kwargs) 2025-10-10T01:01:30.5427711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5427822Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5428143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5428303Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5428307Z 2025-10-10T01:01:30.5428423Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5428650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5428765Z res = mod(**inputs) 2025-10-10T01:01:30.5429068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5429149Z outputs = self.model.decoder( 2025-10-10T01:01:30.5429440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5429530Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5429774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5429872Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5430140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5430218Z return func(*args, **kwargs) 2025-10-10T01:01:30.5430520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5430633Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5430909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5430985Z return func(*args, **kwargs) 2025-10-10T01:01:30.5431276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5431375Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5431379Z 2025-10-10T01:01:30.5431601Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5431835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5431908Z res = mod(**inputs) 2025-10-10T01:01:30.5432211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5432301Z outputs = self.model.decoder( 2025-10-10T01:01:30.5432595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5432686Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5432931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5433029Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5433297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5433372Z return func(*args, **kwargs) 2025-10-10T01:01:30.5433671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5433805Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5433860Z 2025-10-10T01:01:30.5434010Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5434228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5434307Z res = mod(**inputs) 2025-10-10T01:01:30.5434603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5434685Z outputs = self.model.decoder( 2025-10-10T01:01:30.5434995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5435074Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5435327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5435415Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5435708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5435833Z return func(*args, **kwargs) 2025-10-10T01:01:30.5436122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5436262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5436502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5436581Z return self.act(input) 2025-10-10T01:01:30.5436593Z 2025-10-10T01:01:30.5436710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5436928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5437007Z res = mod(**inputs) 2025-10-10T01:01:30.5437314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5437407Z outputs = self.model.decoder( 2025-10-10T01:01:30.5437700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5437780Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5438035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5438123Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5438405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5438481Z return func(*args, **kwargs) 2025-10-10T01:01:30.5438784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5438880Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5438886Z 2025-10-10T01:01:30.5439000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5439225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5439294Z res = mod(**inputs) 2025-10-10T01:01:30.5439586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5439666Z outputs = self.model.decoder( 2025-10-10T01:01:30.5439950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5440036Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5440277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5440370Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5440633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5440746Z return func(*args, **kwargs) 2025-10-10T01:01:30.5441035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5441140Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5441408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5441482Z return func(*args, **kwargs) 2025-10-10T01:01:30.5441762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5441930Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5441934Z 2025-10-10T01:01:30.5442046Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5442282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5442382Z res = mod(**inputs) 2025-10-10T01:01:30.5442676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5442755Z outputs = self.model.decoder( 2025-10-10T01:01:30.5443058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5443146Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5443387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5443480Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5443740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5443816Z return func(*args, **kwargs) 2025-10-10T01:01:30.5444108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5444214Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5444479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5444553Z return func(*args, **kwargs) 2025-10-10T01:01:30.5444845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5444934Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5444938Z 2025-10-10T01:01:30.5445050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5445269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5445340Z res = mod(**inputs) 2025-10-10T01:01:30.5445647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5445731Z outputs = self.model.decoder( 2025-10-10T01:01:30.5446013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5446097Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5446334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5446424Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5446682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5446763Z return func(*args, **kwargs) 2025-10-10T01:01:30.5447044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5447170Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5447457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5447535Z return func(*args, **kwargs) 2025-10-10T01:01:30.5447823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5447918Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5447921Z 2025-10-10T01:01:30.5448011Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5448132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5448344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5448422Z res = mod(**inputs) 2025-10-10T01:01:30.5448710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5448825Z outputs = self.model.decoder( 2025-10-10T01:01:30.5449115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5449193Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5449436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5449522Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5449785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5449860Z return func(*args, **kwargs) 2025-10-10T01:01:30.5450137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5450247Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5450506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5450592Z return func(*args, **kwargs) 2025-10-10T01:01:30.5450870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5450977Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5451295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5451440Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5451444Z 2025-10-10T01:01:30.5451562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5451771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5451848Z res = mod(**inputs) 2025-10-10T01:01:30.5452133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5452215Z outputs = self.model.decoder( 2025-10-10T01:01:30.5452512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5452592Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5452841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5452929Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5453195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5453280Z return func(*args, **kwargs) 2025-10-10T01:01:30.5453567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5453705Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5453992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5454070Z return func(*args, **kwargs) 2025-10-10T01:01:30.5454370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5454461Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5454465Z 2025-10-10T01:01:30.5454589Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5454805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5454883Z res = mod(**inputs) 2025-10-10T01:01:30.5455175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5455257Z outputs = self.model.decoder( 2025-10-10T01:01:30.5455593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5455677Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5455932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5456021Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5456289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5456373Z return func(*args, **kwargs) 2025-10-10T01:01:30.5456662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5456799Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5456803Z 2025-10-10T01:01:30.5456915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5457146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5457220Z res = mod(**inputs) 2025-10-10T01:01:30.5457514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5457605Z outputs = self.model.decoder( 2025-10-10T01:01:30.5457913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5458000Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5458245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5458333Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5458612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5458690Z return func(*args, **kwargs) 2025-10-10T01:01:30.5458991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5459122Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5459362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5459449Z return self.act(input) 2025-10-10T01:01:30.5459453Z 2025-10-10T01:01:30.5459565Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5459791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5459863Z res = mod(**inputs) 2025-10-10T01:01:30.5460174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5460253Z outputs = self.model.decoder( 2025-10-10T01:01:30.5460608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5460701Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5460944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5461039Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5461307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5461384Z return func(*args, **kwargs) 2025-10-10T01:01:30.5461697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5461787Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5461791Z 2025-10-10T01:01:30.5461911Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5462147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5462249Z res = mod(**inputs) 2025-10-10T01:01:30.5462543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5462623Z outputs = self.model.decoder( 2025-10-10T01:01:30.5462932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5463013Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5463267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5463357Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5463625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5463712Z return func(*args, **kwargs) 2025-10-10T01:01:30.5464008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-10-10T01:01:30.5464106Z hidden_states = residual + hidden_states 2025-10-10T01:01:30.5464109Z 2025-10-10T01:01:30.5464224Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5464441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5464522Z res = mod(**inputs) 2025-10-10T01:01:30.5464831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5464922Z outputs = self.model.decoder( 2025-10-10T01:01:30.5465229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5465314Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5465560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5465653Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5465928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5466006Z return func(*args, **kwargs) 2025-10-10T01:01:30.5466300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5466501Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5466774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5466862Z return func(*args, **kwargs) 2025-10-10T01:01:30.5467151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-10-10T01:01:30.5467351Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-10-10T01:01:30.5467357Z 2025-10-10T01:01:30.5467492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5467716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5467790Z res = mod(**inputs) 2025-10-10T01:01:30.5468082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5468172Z outputs = self.model.decoder( 2025-10-10T01:01:30.5468462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5468550Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5468792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5468879Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5469207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5469286Z return func(*args, **kwargs) 2025-10-10T01:01:30.5469585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5469692Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5469945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5470026Z return func(*args, **kwargs) 2025-10-10T01:01:30.5470304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-10-10T01:01:30.5470399Z key_states = self.k_proj(current_states) 2025-10-10T01:01:30.5470402Z 2025-10-10T01:01:30.5470512Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5470736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5470806Z res = mod(**inputs) 2025-10-10T01:01:30.5471087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5471172Z outputs = self.model.decoder( 2025-10-10T01:01:30.5471453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5471537Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5471774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5471858Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5472121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5472200Z return func(*args, **kwargs) 2025-10-10T01:01:30.5472485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5472589Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5472854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5472928Z return func(*args, **kwargs) 2025-10-10T01:01:30.5473205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-10-10T01:01:30.5473308Z value_states = self.v_proj(current_states) 2025-10-10T01:01:30.5473312Z 2025-10-10T01:01:30.5473401Z cudagraph partition due to non gpu ops 2025-10-10T01:01:30.5473520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5473733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5473827Z res = mod(**inputs) 2025-10-10T01:01:30.5474137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5474219Z outputs = self.model.decoder( 2025-10-10T01:01:30.5474510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5474590Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5474829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5474922Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5475180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5475262Z return func(*args, **kwargs) 2025-10-10T01:01:30.5475556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5475690Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5475997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5476305Z return func(*args, **kwargs) 2025-10-10T01:01:30.5476610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-10-10T01:01:30.5476781Z attn_output, attn_weights = attention_interface( 2025-10-10T01:01:30.5477121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-10-10T01:01:30.5477322Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:30.5477326Z 2025-10-10T01:01:30.5477499Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5477757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5477891Z res = mod(**inputs) 2025-10-10T01:01:30.5478198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5478331Z outputs = self.model.decoder( 2025-10-10T01:01:30.5478628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5478754Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5479066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5479177Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5479488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5479591Z return func(*args, **kwargs) 2025-10-10T01:01:30.5479938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-10-10T01:01:30.5480099Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:01:30.5480387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5480514Z return func(*args, **kwargs) 2025-10-10T01:01:30.5480839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-10-10T01:01:30.5480968Z attn_output = self.out_proj(attn_output) 2025-10-10T01:01:30.5480972Z 2025-10-10T01:01:30.5481128Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5481414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5481524Z res = mod(**inputs) 2025-10-10T01:01:30.5482624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5482770Z outputs = self.model.decoder( 2025-10-10T01:01:30.5483063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5483238Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5483504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5483613Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5483925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5484026Z return func(*args, **kwargs) 2025-10-10T01:01:30.5484383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5484587Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5484591Z 2025-10-10T01:01:30.5484753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5484990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5485083Z res = mod(**inputs) 2025-10-10T01:01:30.5485446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5485569Z outputs = self.model.decoder( 2025-10-10T01:01:30.5485912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5486014Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5486277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5486428Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5486719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5486888Z return func(*args, **kwargs) 2025-10-10T01:01:30.5487204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-10-10T01:01:30.5487385Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:01:30.5487655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:30.5487757Z return self.act(input) 2025-10-10T01:01:30.5487761Z 2025-10-10T01:01:30.5487949Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5488202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5488323Z res = mod(**inputs) 2025-10-10T01:01:30.5488650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-10-10T01:01:30.5488785Z outputs = self.model.decoder( 2025-10-10T01:01:30.5489086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-10-10T01:01:30.5489210Z layer_outputs = decoder_layer( 2025-10-10T01:01:30.5489513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:30.5489623Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:30.5489951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:30.5490068Z return func(*args, **kwargs) 2025-10-10T01:01:30.5490360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-10-10T01:01:30.5490553Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:01:30.5490573Z 2025-10-10T01:01:30.5490718Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5490984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5491081Z res = mod(**inputs) 2025-10-10T01:01:30.5491407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1651, in forward 2025-10-10T01:01:30.5491538Z logits = self.lm_head(outputs[0]) 2025-10-10T01:01:30.5491543Z 2025-10-10T01:01:30.5491689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:30.5491967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:30.5492062Z res = mod(**inputs) 2025-10-10T01:01:30.5492400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1657, in forward 2025-10-10T01:01:30.5492608Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T01:01:30.5492613Z 2025-10-10T01:01:41.2543229Z Compilation time (from dynamo_timed): 17.41265092 2025-10-10T01:01:41.2563019Z pass 2025-10-10T01:01:41.2563518Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:41.2570135Z TIMING: _recursive_pre_grad_passes:0.00748 _recursive_joint_graph_passes:0.70148 _recursive_post_grad_passes:0.07697 async_compile.wait:0.83122 code_gen:9.92869 inductor_compile:11.30042 backend_compile:14.63559 gc:0.00087 entire_frame_compile:17.41265 total_wall_time:17.41265 2025-10-10T01:01:41.2571597Z STATS: call_* op count: 333 | FakeTensorMode.__torch_dispatch__:7436 | FakeTensor.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:2000 2025-10-10T01:01:41.2576421Z Dynamo produced 1 graphs covering 333 ops with 0 graph breaks (0 unique) 2025-10-10T01:01:44.0567325Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:01:44.0568322Z import pynvml # type: ignore[import] 2025-10-10T01:01:47.6099942Z 2025-10-10T01:01:47.6221160Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-10-10T01:01:47.6222010Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-10-10T01:01:48.9195637Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-10-10T01:01:48.9197257Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-10-10T01:01:48.9199094Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-10-10T01:01:48.9200721Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-10-10T01:01:49.0934727Z 2025-10-10T01:01:49.0935705Z loading model: 0it [00:01, ?it/s] 2025-10-10T01:01:49.0947195Z cpu eval RobertaForCausalLM 2025-10-10T01:01:49.6992704Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:49.9921905Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:50.2935398Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:01:58.3907743Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3910626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3911210Z res = mod(**inputs) 2025-10-10T01:01:58.3911798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3912378Z outputs = self.roberta( 2025-10-10T01:01:58.3913434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-10-10T01:01:58.3914121Z embedding_output = self.embeddings( 2025-10-10T01:01:58.3915128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-10-10T01:01:58.3926026Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-10-10T01:01:58.3926994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1547, in create_position_ids_from_input_ids 2025-10-10T01:01:58.3927539Z mask = input_ids.ne(padding_idx).int() 2025-10-10T01:01:58.3927714Z 2025-10-10T01:01:58.3927814Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3928157Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3928395Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3928624Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3928859Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3929100Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3929335Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3929560Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3929793Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3930036Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3930272Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3930503Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3930768Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3931202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3931919Z res = mod(**inputs) 2025-10-10T01:01:58.3932374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3932841Z outputs = self.roberta( 2025-10-10T01:01:58.3933279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-10-10T01:01:58.3933748Z embedding_output = self.embeddings( 2025-10-10T01:01:58.3934203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-10-10T01:01:58.3934804Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-10-10T01:01:58.3935464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-10-10T01:01:58.3936113Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T01:01:58.3936381Z 2025-10-10T01:01:58.3936503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3936912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3937292Z res = mod(**inputs) 2025-10-10T01:01:58.3937731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3938184Z outputs = self.roberta( 2025-10-10T01:01:58.3938767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-10-10T01:01:58.3939208Z embedding_output = self.embeddings( 2025-10-10T01:01:58.3939661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-10-10T01:01:58.3940218Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-10-10T01:01:58.3940863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-10-10T01:01:58.3941494Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-10-10T01:01:58.3941754Z 2025-10-10T01:01:58.3941885Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3942293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3942722Z res = mod(**inputs) 2025-10-10T01:01:58.3943141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3943583Z outputs = self.roberta( 2025-10-10T01:01:58.3944015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3944465Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3944924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3945373Z layer_outputs = layer_module( 2025-10-10T01:01:58.3945777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3946196Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3946814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3947260Z return func(*args, **kwargs) 2025-10-10T01:01:58.3947697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.3948161Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.3948601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3949016Z return func(*args, **kwargs) 2025-10-10T01:01:58.3949446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.3949892Z self_outputs = self.self( 2025-10-10T01:01:58.3950305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3950727Z return func(*args, **kwargs) 2025-10-10T01:01:58.3951161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.3951762Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.3952063Z 2025-10-10T01:01:58.3952186Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3952595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3952951Z res = mod(**inputs) 2025-10-10T01:01:58.3953370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3953809Z outputs = self.roberta( 2025-10-10T01:01:58.3954231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3954705Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3955176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3955625Z layer_outputs = layer_module( 2025-10-10T01:01:58.3956025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3956441Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3956879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3957292Z return func(*args, **kwargs) 2025-10-10T01:01:58.3957728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.3958199Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.3958655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3959083Z return func(*args, **kwargs) 2025-10-10T01:01:58.3959516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.3959962Z self_outputs = self.self( 2025-10-10T01:01:58.3960369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3960789Z return func(*args, **kwargs) 2025-10-10T01:01:58.3961213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.3961650Z self.key(current_states) 2025-10-10T01:01:58.3961787Z 2025-10-10T01:01:58.3961908Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3962318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3962684Z res = mod(**inputs) 2025-10-10T01:01:58.3963092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3963537Z outputs = self.roberta( 2025-10-10T01:01:58.3963950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3964376Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3964794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3965223Z layer_outputs = layer_module( 2025-10-10T01:01:58.3965603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3966000Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3966417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3966818Z return func(*args, **kwargs) 2025-10-10T01:01:58.3967235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.3967695Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.3968128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3968523Z return func(*args, **kwargs) 2025-10-10T01:01:58.3968943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.3969372Z self_outputs = self.self( 2025-10-10T01:01:58.3969771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3970187Z return func(*args, **kwargs) 2025-10-10T01:01:58.3970648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.3971092Z self.value(current_states) 2025-10-10T01:01:58.3971234Z 2025-10-10T01:01:58.3971327Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.3971615Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3972010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3972353Z res = mod(**inputs) 2025-10-10T01:01:58.3972757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3973179Z outputs = self.roberta( 2025-10-10T01:01:58.3973585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3974004Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3974451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3974892Z layer_outputs = layer_module( 2025-10-10T01:01:58.3975274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3975670Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3976077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3976482Z return func(*args, **kwargs) 2025-10-10T01:01:58.3976899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.3977339Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.3977749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3978158Z return func(*args, **kwargs) 2025-10-10T01:01:58.3978580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.3979022Z self_outputs = self.self( 2025-10-10T01:01:58.3979433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3979826Z return func(*args, **kwargs) 2025-10-10T01:01:58.3980265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.3980764Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.3980966Z 2025-10-10T01:01:58.3981096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3981503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3981861Z res = mod(**inputs) 2025-10-10T01:01:58.3982284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3982720Z outputs = self.roberta( 2025-10-10T01:01:58.3983148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3983589Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3984040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3984480Z layer_outputs = layer_module( 2025-10-10T01:01:58.3984875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3985277Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3985696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3986140Z return func(*args, **kwargs) 2025-10-10T01:01:58.3986696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.3987160Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.3987593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3988000Z return func(*args, **kwargs) 2025-10-10T01:01:58.3988443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.3988957Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.3989466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.3989931Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.3990112Z 2025-10-10T01:01:58.3990255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3990664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3991026Z res = mod(**inputs) 2025-10-10T01:01:58.3991435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.3991863Z outputs = self.roberta( 2025-10-10T01:01:58.3992278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.3992716Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.3993143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.3993585Z layer_outputs = layer_module( 2025-10-10T01:01:58.3993982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.3994394Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.3994811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.3995233Z return func(*args, **kwargs) 2025-10-10T01:01:58.3995661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.3996104Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.3996589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.3997010Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.3997473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.3997994Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.3998477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.3998914Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.3999067Z 2025-10-10T01:01:58.3999182Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.3999574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.3999924Z res = mod(**inputs) 2025-10-10T01:01:58.4000317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4000730Z outputs = self.roberta( 2025-10-10T01:01:58.4001135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4001586Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4002035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4002459Z layer_outputs = layer_module( 2025-10-10T01:01:58.4002839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4003250Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4003666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4004078Z return func(*args, **kwargs) 2025-10-10T01:01:58.4004521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4004960Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4005411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4005864Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4006328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4006838Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4007312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4007785Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4008204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4008581Z return self.act(input) 2025-10-10T01:01:58.4008706Z 2025-10-10T01:01:58.4008822Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4009222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4009581Z res = mod(**inputs) 2025-10-10T01:01:58.4010000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4010429Z outputs = self.roberta( 2025-10-10T01:01:58.4010855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4011307Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4011754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4012206Z layer_outputs = layer_module( 2025-10-10T01:01:58.4012592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4013032Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4013475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4013908Z return func(*args, **kwargs) 2025-10-10T01:01:58.4014352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4014865Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4015317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4015760Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4016241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4016792Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4017307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4017823Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4017988Z 2025-10-10T01:01:58.4018108Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4018510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4018869Z res = mod(**inputs) 2025-10-10T01:01:58.4019281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4019714Z outputs = self.roberta( 2025-10-10T01:01:58.4020139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4020582Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4021016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4021493Z layer_outputs = layer_module( 2025-10-10T01:01:58.4021887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4022295Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4022718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4023127Z return func(*args, **kwargs) 2025-10-10T01:01:58.4023568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4024028Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4024453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4024866Z return func(*args, **kwargs) 2025-10-10T01:01:58.4025300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4025743Z self_outputs = self.self( 2025-10-10T01:01:58.4026146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4026657Z return func(*args, **kwargs) 2025-10-10T01:01:58.4027093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4027696Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4028000Z 2025-10-10T01:01:58.4028121Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4028534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4028896Z res = mod(**inputs) 2025-10-10T01:01:58.4029321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4029769Z outputs = self.roberta( 2025-10-10T01:01:58.4030193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4030648Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4031102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4031674Z layer_outputs = layer_module( 2025-10-10T01:01:58.4032084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4032512Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4032939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4033350Z return func(*args, **kwargs) 2025-10-10T01:01:58.4033865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4034303Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4034720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4035123Z return func(*args, **kwargs) 2025-10-10T01:01:58.4035531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4035961Z self_outputs = self.self( 2025-10-10T01:01:58.4036355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4036758Z return func(*args, **kwargs) 2025-10-10T01:01:58.4037162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4037644Z self.key(current_states) 2025-10-10T01:01:58.4037779Z 2025-10-10T01:01:58.4037895Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4038283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4038632Z res = mod(**inputs) 2025-10-10T01:01:58.4039021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4039446Z outputs = self.roberta( 2025-10-10T01:01:58.4039854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4040284Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4040710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4041128Z layer_outputs = layer_module( 2025-10-10T01:01:58.4041516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4041913Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4042326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4042721Z return func(*args, **kwargs) 2025-10-10T01:01:58.4043151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4043595Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4044009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4044410Z return func(*args, **kwargs) 2025-10-10T01:01:58.4044834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4045263Z self_outputs = self.self( 2025-10-10T01:01:58.4045655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4046038Z return func(*args, **kwargs) 2025-10-10T01:01:58.4046426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4046826Z self.value(current_states) 2025-10-10T01:01:58.4046957Z 2025-10-10T01:01:58.4047042Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4047296Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4047664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4048017Z res = mod(**inputs) 2025-10-10T01:01:58.4048398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4048818Z outputs = self.roberta( 2025-10-10T01:01:58.4049234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4049639Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4050035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4050463Z layer_outputs = layer_module( 2025-10-10T01:01:58.4050825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4051206Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4051610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4052000Z return func(*args, **kwargs) 2025-10-10T01:01:58.4052435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4052869Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4053265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4053638Z return func(*args, **kwargs) 2025-10-10T01:01:58.4054056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4054510Z self_outputs = self.self( 2025-10-10T01:01:58.4054882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4055266Z return func(*args, **kwargs) 2025-10-10T01:01:58.4055658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4056134Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4056333Z 2025-10-10T01:01:58.4056446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4056818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4057141Z res = mod(**inputs) 2025-10-10T01:01:58.4057524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4057936Z outputs = self.roberta( 2025-10-10T01:01:58.4058326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4058748Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4059142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4059550Z layer_outputs = layer_module( 2025-10-10T01:01:58.4059916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4060283Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4060669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4061104Z return func(*args, **kwargs) 2025-10-10T01:01:58.4061529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4061980Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4062400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4062776Z return func(*args, **kwargs) 2025-10-10T01:01:58.4063172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4063652Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4064129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4064566Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4064718Z 2025-10-10T01:01:58.4064832Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4065226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4065577Z res = mod(**inputs) 2025-10-10T01:01:58.4065991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4066493Z outputs = self.roberta( 2025-10-10T01:01:58.4066934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4067435Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4067842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4068236Z layer_outputs = layer_module( 2025-10-10T01:01:58.4068582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4068949Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4069330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4069716Z return func(*args, **kwargs) 2025-10-10T01:01:58.4070099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4070500Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4070903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4071301Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4071731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4072204Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4072636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4073042Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4073186Z 2025-10-10T01:01:58.4073292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4073654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4073980Z res = mod(**inputs) 2025-10-10T01:01:58.4074344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4074739Z outputs = self.roberta( 2025-10-10T01:01:58.4075115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4075505Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4075885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4076276Z layer_outputs = layer_module( 2025-10-10T01:01:58.4076625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4076992Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4077368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4077742Z return func(*args, **kwargs) 2025-10-10T01:01:58.4078149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4078546Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4078947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4079336Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4079755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4080220Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4080653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4081084Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4081487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4081844Z return self.act(input) 2025-10-10T01:01:58.4081965Z 2025-10-10T01:01:58.4082072Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4082434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4082757Z res = mod(**inputs) 2025-10-10T01:01:58.4083118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4083521Z outputs = self.roberta( 2025-10-10T01:01:58.4083904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4084308Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4084709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4085119Z layer_outputs = layer_module( 2025-10-10T01:01:58.4085470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4085838Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4086222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4086590Z return func(*args, **kwargs) 2025-10-10T01:01:58.4086975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4087381Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4087782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4088174Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4088595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4089079Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4089532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4089946Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4090090Z 2025-10-10T01:01:58.4090206Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4090568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4090899Z res = mod(**inputs) 2025-10-10T01:01:58.4091276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4091671Z outputs = self.roberta( 2025-10-10T01:01:58.4092050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4092505Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4092903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4093306Z layer_outputs = layer_module( 2025-10-10T01:01:58.4093665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4094050Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4094464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4094867Z return func(*args, **kwargs) 2025-10-10T01:01:58.4095296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4095753Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4096161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4096539Z return func(*args, **kwargs) 2025-10-10T01:01:58.4096929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4097330Z self_outputs = self.self( 2025-10-10T01:01:58.4097694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4098093Z return func(*args, **kwargs) 2025-10-10T01:01:58.4098514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4099079Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4099352Z 2025-10-10T01:01:58.4099472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4099844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4100188Z res = mod(**inputs) 2025-10-10T01:01:58.4100598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4101030Z outputs = self.roberta( 2025-10-10T01:01:58.4101448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4101878Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4102313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4102747Z layer_outputs = layer_module( 2025-10-10T01:01:58.4103123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4103516Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4103927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4104328Z return func(*args, **kwargs) 2025-10-10T01:01:58.4104764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4105210Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4105616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4106030Z return func(*args, **kwargs) 2025-10-10T01:01:58.4106526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4106994Z self_outputs = self.self( 2025-10-10T01:01:58.4107465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4107859Z return func(*args, **kwargs) 2025-10-10T01:01:58.4108268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4108713Z self.key(current_states) 2025-10-10T01:01:58.4108832Z 2025-10-10T01:01:58.4109241Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4109611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4109944Z res = mod(**inputs) 2025-10-10T01:01:58.4110347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4110782Z outputs = self.roberta( 2025-10-10T01:01:58.4111220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4111672Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4112097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4112524Z layer_outputs = layer_module( 2025-10-10T01:01:58.4112899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4113292Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4113698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4114111Z return func(*args, **kwargs) 2025-10-10T01:01:58.4114521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4114960Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4115375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4115780Z return func(*args, **kwargs) 2025-10-10T01:01:58.4116192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4116617Z self_outputs = self.self( 2025-10-10T01:01:58.4117005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4117399Z return func(*args, **kwargs) 2025-10-10T01:01:58.4117814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4118239Z self.value(current_states) 2025-10-10T01:01:58.4118368Z 2025-10-10T01:01:58.4118467Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4118731Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4119121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4119471Z res = mod(**inputs) 2025-10-10T01:01:58.4119877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4120301Z outputs = self.roberta( 2025-10-10T01:01:58.4120702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4121130Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4121551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4121990Z layer_outputs = layer_module( 2025-10-10T01:01:58.4122352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4122743Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4123152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4123556Z return func(*args, **kwargs) 2025-10-10T01:01:58.4123969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4124399Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4124819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4125223Z return func(*args, **kwargs) 2025-10-10T01:01:58.4125633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4126054Z self_outputs = self.self( 2025-10-10T01:01:58.4126455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4126883Z return func(*args, **kwargs) 2025-10-10T01:01:58.4127298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4127795Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4127985Z 2025-10-10T01:01:58.4128106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4128486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4128842Z res = mod(**inputs) 2025-10-10T01:01:58.4129246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4129670Z outputs = self.roberta( 2025-10-10T01:01:58.4130076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4130487Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4130890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4131293Z layer_outputs = layer_module( 2025-10-10T01:01:58.4131771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4132146Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4132541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4132924Z return func(*args, **kwargs) 2025-10-10T01:01:58.4133317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4133730Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4134124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4134505Z return func(*args, **kwargs) 2025-10-10T01:01:58.4134899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4135359Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4135810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4136229Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4136380Z 2025-10-10T01:01:58.4136488Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4136858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4137186Z res = mod(**inputs) 2025-10-10T01:01:58.4137560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4138049Z outputs = self.roberta( 2025-10-10T01:01:58.4138432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4138837Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4139215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4139596Z layer_outputs = layer_module( 2025-10-10T01:01:58.4139946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4140337Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4140750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4141151Z return func(*args, **kwargs) 2025-10-10T01:01:58.4141663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4142109Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4142529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4142938Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4143372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4143871Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4144354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4144792Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4144944Z 2025-10-10T01:01:58.4145068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4145452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4145805Z res = mod(**inputs) 2025-10-10T01:01:58.4146203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4146705Z outputs = self.roberta( 2025-10-10T01:01:58.4147122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4147551Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4147957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4148339Z layer_outputs = layer_module( 2025-10-10T01:01:58.4148684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4149037Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4149413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4149776Z return func(*args, **kwargs) 2025-10-10T01:01:58.4150150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4150546Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4150930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4151311Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4151722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4152181Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4152660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4153069Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4153439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4153778Z return self.act(input) 2025-10-10T01:01:58.4153890Z 2025-10-10T01:01:58.4154002Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4154355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4154679Z res = mod(**inputs) 2025-10-10T01:01:58.4155045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4155433Z outputs = self.roberta( 2025-10-10T01:01:58.4155831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4156223Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4156613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4157005Z layer_outputs = layer_module( 2025-10-10T01:01:58.4157352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4157724Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4158101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4158474Z return func(*args, **kwargs) 2025-10-10T01:01:58.4158860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4159274Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4159671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4160067Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4160490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4160975Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4161425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4161821Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4161970Z 2025-10-10T01:01:58.4162075Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4162439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4162769Z res = mod(**inputs) 2025-10-10T01:01:58.4163141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4163533Z outputs = self.roberta( 2025-10-10T01:01:58.4163910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4164304Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4164692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4165075Z layer_outputs = layer_module( 2025-10-10T01:01:58.4165421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4165783Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4166169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4166577Z return func(*args, **kwargs) 2025-10-10T01:01:58.4166957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4167365Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4167800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4168180Z return func(*args, **kwargs) 2025-10-10T01:01:58.4168559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4168945Z self_outputs = self.self( 2025-10-10T01:01:58.4169307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4169676Z return func(*args, **kwargs) 2025-10-10T01:01:58.4170094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4170615Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4170884Z 2025-10-10T01:01:58.4170989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4171349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4171670Z res = mod(**inputs) 2025-10-10T01:01:58.4172038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4172419Z outputs = self.roberta( 2025-10-10T01:01:58.4172794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4173201Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4173608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4174016Z layer_outputs = layer_module( 2025-10-10T01:01:58.4174356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4174722Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4175101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4175473Z return func(*args, **kwargs) 2025-10-10T01:01:58.4175836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4176229Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4176600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4176971Z return func(*args, **kwargs) 2025-10-10T01:01:58.4177351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4177732Z self_outputs = self.self( 2025-10-10T01:01:58.4178092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4178464Z return func(*args, **kwargs) 2025-10-10T01:01:58.4178843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4179229Z self.key(current_states) 2025-10-10T01:01:58.4179343Z 2025-10-10T01:01:58.4179447Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4179808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4180142Z res = mod(**inputs) 2025-10-10T01:01:58.4180522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4180893Z outputs = self.roberta( 2025-10-10T01:01:58.4181263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4181655Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4182055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4182455Z layer_outputs = layer_module( 2025-10-10T01:01:58.4182803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4183177Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4183564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4183980Z return func(*args, **kwargs) 2025-10-10T01:01:58.4184389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4184823Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4185239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4185643Z return func(*args, **kwargs) 2025-10-10T01:01:58.4186059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4186558Z self_outputs = self.self( 2025-10-10T01:01:58.4186953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4187356Z return func(*args, **kwargs) 2025-10-10T01:01:58.4187776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4188181Z self.value(current_states) 2025-10-10T01:01:58.4188303Z 2025-10-10T01:01:58.4188391Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4188646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4189017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4189351Z res = mod(**inputs) 2025-10-10T01:01:58.4189736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4190141Z outputs = self.roberta( 2025-10-10T01:01:58.4190550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4190980Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4191408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4191817Z layer_outputs = layer_module( 2025-10-10T01:01:58.4192178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4192554Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4192947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4193332Z return func(*args, **kwargs) 2025-10-10T01:01:58.4193716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4194130Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4194524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4194929Z return func(*args, **kwargs) 2025-10-10T01:01:58.4195327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4195730Z self_outputs = self.self( 2025-10-10T01:01:58.4196106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4196494Z return func(*args, **kwargs) 2025-10-10T01:01:58.4196885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4197354Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4197553Z 2025-10-10T01:01:58.4197663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4198037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4198371Z res = mod(**inputs) 2025-10-10T01:01:58.4198797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4199188Z outputs = self.roberta( 2025-10-10T01:01:58.4199571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4199974Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4200367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4200759Z layer_outputs = layer_module( 2025-10-10T01:01:58.4201115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4201489Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4201880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4202265Z return func(*args, **kwargs) 2025-10-10T01:01:58.4202654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4203072Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4203464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4203865Z return func(*args, **kwargs) 2025-10-10T01:01:58.4204276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4204761Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4205241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4205690Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4205837Z 2025-10-10T01:01:58.4205954Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4206323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4206672Z res = mod(**inputs) 2025-10-10T01:01:58.4207066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4207486Z outputs = self.roberta( 2025-10-10T01:01:58.4207894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4208312Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4208731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4209153Z layer_outputs = layer_module( 2025-10-10T01:01:58.4209533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4209968Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4210380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4210793Z return func(*args, **kwargs) 2025-10-10T01:01:58.4211235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4211686Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4212120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4212554Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4213024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4213574Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4214064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4214498Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4214658Z 2025-10-10T01:01:58.4214772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4215165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4215518Z res = mod(**inputs) 2025-10-10T01:01:58.4215917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4216341Z outputs = self.roberta( 2025-10-10T01:01:58.4216747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4217150Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4217551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4217946Z layer_outputs = layer_module( 2025-10-10T01:01:58.4218305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4218680Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4219081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4219495Z return func(*args, **kwargs) 2025-10-10T01:01:58.4219922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4220363Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4220802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4221233Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4221691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4222204Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4222707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4223185Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4223626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4224020Z return self.act(input) 2025-10-10T01:01:58.4224147Z 2025-10-10T01:01:58.4224264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4224683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4225070Z res = mod(**inputs) 2025-10-10T01:01:58.4225491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4225904Z outputs = self.roberta( 2025-10-10T01:01:58.4226309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4226822Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4227258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4227702Z layer_outputs = layer_module( 2025-10-10T01:01:58.4228052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4228436Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4228886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4229321Z return func(*args, **kwargs) 2025-10-10T01:01:58.4229762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4230182Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4230597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4230989Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4231418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4232036Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4232510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4232942Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4233084Z 2025-10-10T01:01:58.4233204Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4233568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4233891Z res = mod(**inputs) 2025-10-10T01:01:58.4234287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4234716Z outputs = self.roberta( 2025-10-10T01:01:58.4235128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4235556Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4235984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4236398Z layer_outputs = layer_module( 2025-10-10T01:01:58.4236775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4237175Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4237585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4238013Z return func(*args, **kwargs) 2025-10-10T01:01:58.4238430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4238875Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4239296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4239706Z return func(*args, **kwargs) 2025-10-10T01:01:58.4240141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4240649Z self_outputs = self.self( 2025-10-10T01:01:58.4241040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4241442Z return func(*args, **kwargs) 2025-10-10T01:01:58.4241844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4242417Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4242708Z 2025-10-10T01:01:58.4242824Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4243215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4243557Z res = mod(**inputs) 2025-10-10T01:01:58.4244002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4244459Z outputs = self.roberta( 2025-10-10T01:01:58.4244864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4245296Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4245712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4246139Z layer_outputs = layer_module( 2025-10-10T01:01:58.4246514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4246911Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4247323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4247719Z return func(*args, **kwargs) 2025-10-10T01:01:58.4248140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4248580Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4249000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4249396Z return func(*args, **kwargs) 2025-10-10T01:01:58.4249795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4250184Z self_outputs = self.self( 2025-10-10T01:01:58.4250560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4250939Z return func(*args, **kwargs) 2025-10-10T01:01:58.4251324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4251730Z self.key(current_states) 2025-10-10T01:01:58.4251857Z 2025-10-10T01:01:58.4251979Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4252342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4252661Z res = mod(**inputs) 2025-10-10T01:01:58.4253019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4253409Z outputs = self.roberta( 2025-10-10T01:01:58.4253781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4254199Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4254615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4255039Z layer_outputs = layer_module( 2025-10-10T01:01:58.4255461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4255861Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4256275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4256664Z return func(*args, **kwargs) 2025-10-10T01:01:58.4257043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4257446Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4257826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4258189Z return func(*args, **kwargs) 2025-10-10T01:01:58.4258582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4259037Z self_outputs = self.self( 2025-10-10T01:01:58.4259410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4259794Z return func(*args, **kwargs) 2025-10-10T01:01:58.4260192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4260621Z self.value(current_states) 2025-10-10T01:01:58.4260756Z 2025-10-10T01:01:58.4260849Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4261113Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4261515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4261889Z res = mod(**inputs) 2025-10-10T01:01:58.4262308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4262743Z outputs = self.roberta( 2025-10-10T01:01:58.4263154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4263577Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4264003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4264436Z layer_outputs = layer_module( 2025-10-10T01:01:58.4264823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4265223Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4265643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4266064Z return func(*args, **kwargs) 2025-10-10T01:01:58.4266558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4267014Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4267444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4267871Z return func(*args, **kwargs) 2025-10-10T01:01:58.4268296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4268735Z self_outputs = self.self( 2025-10-10T01:01:58.4269132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4269536Z return func(*args, **kwargs) 2025-10-10T01:01:58.4269952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4270445Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4270673Z 2025-10-10T01:01:58.4270814Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4271205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4271560Z res = mod(**inputs) 2025-10-10T01:01:58.4271959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4272382Z outputs = self.roberta( 2025-10-10T01:01:58.4272790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4273209Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4273629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4274052Z layer_outputs = layer_module( 2025-10-10T01:01:58.4274481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4274875Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4275275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4275676Z return func(*args, **kwargs) 2025-10-10T01:01:58.4276064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4276479Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4276855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4277215Z return func(*args, **kwargs) 2025-10-10T01:01:58.4277592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4278042Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4278479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4278878Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4279015Z 2025-10-10T01:01:58.4279119Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4279472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4279796Z res = mod(**inputs) 2025-10-10T01:01:58.4280169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4280558Z outputs = self.roberta( 2025-10-10T01:01:58.4280941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4281347Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4281748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4282173Z layer_outputs = layer_module( 2025-10-10T01:01:58.4282541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4282931Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4283351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4283765Z return func(*args, **kwargs) 2025-10-10T01:01:58.4284171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4284616Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4285050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4285527Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4285967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4286445Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4286901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4287315Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4287461Z 2025-10-10T01:01:58.4287584Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4287973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4288329Z res = mod(**inputs) 2025-10-10T01:01:58.4288752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4289194Z outputs = self.roberta( 2025-10-10T01:01:58.4289597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4290021Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4290432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4290857Z layer_outputs = layer_module( 2025-10-10T01:01:58.4291231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4291627Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4292040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4292450Z return func(*args, **kwargs) 2025-10-10T01:01:58.4292867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4293310Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4293740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4294155Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4294614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4295125Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4295603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4296072Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4296481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4296857Z return self.act(input) 2025-10-10T01:01:58.4296985Z 2025-10-10T01:01:58.4297101Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4297490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4297833Z res = mod(**inputs) 2025-10-10T01:01:58.4298225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4298646Z outputs = self.roberta( 2025-10-10T01:01:58.4299061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4299495Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4299917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4300381Z layer_outputs = layer_module( 2025-10-10T01:01:58.4300787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4301197Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4301631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4302057Z return func(*args, **kwargs) 2025-10-10T01:01:58.4302491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4302949Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4303402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4303838Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4304337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4304895Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4305400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4305851Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4306007Z 2025-10-10T01:01:58.4306125Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4306620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4306993Z res = mod(**inputs) 2025-10-10T01:01:58.4307407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4307846Z outputs = self.roberta( 2025-10-10T01:01:58.4308274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4308713Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4309150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4309632Z layer_outputs = layer_module( 2025-10-10T01:01:58.4310014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4310426Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4310859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4311289Z return func(*args, **kwargs) 2025-10-10T01:01:58.4311715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4312163Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4312609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4313031Z return func(*args, **kwargs) 2025-10-10T01:01:58.4313456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4313891Z self_outputs = self.self( 2025-10-10T01:01:58.4314287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4314699Z return func(*args, **kwargs) 2025-10-10T01:01:58.4315122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4315715Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4316034Z 2025-10-10T01:01:58.4316164Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4316592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4316950Z res = mod(**inputs) 2025-10-10T01:01:58.4317358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4317792Z outputs = self.roberta( 2025-10-10T01:01:58.4318203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4318641Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4319066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4319501Z layer_outputs = layer_module( 2025-10-10T01:01:58.4319889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4320330Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4320760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4321172Z return func(*args, **kwargs) 2025-10-10T01:01:58.4321596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4322034Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4322461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4322880Z return func(*args, **kwargs) 2025-10-10T01:01:58.4323302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4323737Z self_outputs = self.self( 2025-10-10T01:01:58.4324134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4324548Z return func(*args, **kwargs) 2025-10-10T01:01:58.4324973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4325411Z self.key(current_states) 2025-10-10T01:01:58.4325538Z 2025-10-10T01:01:58.4325661Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4326055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4326412Z res = mod(**inputs) 2025-10-10T01:01:58.4326817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4327252Z outputs = self.roberta( 2025-10-10T01:01:58.4327661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4328104Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4328533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4328967Z layer_outputs = layer_module( 2025-10-10T01:01:58.4329354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4329752Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4330172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4330587Z return func(*args, **kwargs) 2025-10-10T01:01:58.4331008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4331559Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4332083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4332500Z return func(*args, **kwargs) 2025-10-10T01:01:58.4332928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4333364Z self_outputs = self.self( 2025-10-10T01:01:58.4333750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4334156Z return func(*args, **kwargs) 2025-10-10T01:01:58.4334567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4334992Z self.value(current_states) 2025-10-10T01:01:58.4335121Z 2025-10-10T01:01:58.4335219Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4335475Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4336083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4336437Z res = mod(**inputs) 2025-10-10T01:01:58.4336836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4337264Z outputs = self.roberta( 2025-10-10T01:01:58.4337667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4338098Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4338516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4338951Z layer_outputs = layer_module( 2025-10-10T01:01:58.4339320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4339722Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4340137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4340542Z return func(*args, **kwargs) 2025-10-10T01:01:58.4340973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4341398Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4341667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4341743Z return func(*args, **kwargs) 2025-10-10T01:01:58.4342048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4342128Z self_outputs = self.self( 2025-10-10T01:01:58.4342389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4342479Z return func(*args, **kwargs) 2025-10-10T01:01:58.4342764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4342920Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4342924Z 2025-10-10T01:01:58.4343043Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4343260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4343341Z res = mod(**inputs) 2025-10-10T01:01:58.4343631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4343719Z outputs = self.roberta( 2025-10-10T01:01:58.4344003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4344115Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4344419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4344504Z layer_outputs = layer_module( 2025-10-10T01:01:58.4344760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4344852Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4345125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4345204Z return func(*args, **kwargs) 2025-10-10T01:01:58.4345492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4345592Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4345881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4345990Z return func(*args, **kwargs) 2025-10-10T01:01:58.4346275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4346473Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4346774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4346869Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4346874Z 2025-10-10T01:01:58.4347000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4347219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4347309Z res = mod(**inputs) 2025-10-10T01:01:58.4347593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4347674Z outputs = self.roberta( 2025-10-10T01:01:58.4347961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4348042Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4348329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4348410Z layer_outputs = layer_module( 2025-10-10T01:01:58.4348650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4348746Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4349006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4349091Z return func(*args, **kwargs) 2025-10-10T01:01:58.4349378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4349481Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4349763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4349850Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4350179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4350313Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4350604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4350694Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4350698Z 2025-10-10T01:01:58.4350836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4351074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4351140Z res = mod(**inputs) 2025-10-10T01:01:58.4351402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4351472Z outputs = self.roberta( 2025-10-10T01:01:58.4351743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4351815Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4352061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4352138Z layer_outputs = layer_module( 2025-10-10T01:01:58.4352352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4352470Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4352705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4352774Z return func(*args, **kwargs) 2025-10-10T01:01:58.4353031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4353113Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4353364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4353440Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4353729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4353856Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4354115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4354243Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4354451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4354526Z return self.act(input) 2025-10-10T01:01:58.4354529Z 2025-10-10T01:01:58.4354632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4354819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4354891Z res = mod(**inputs) 2025-10-10T01:01:58.4355138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4355214Z outputs = self.roberta( 2025-10-10T01:01:58.4355466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4355539Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4355792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4355863Z layer_outputs = layer_module( 2025-10-10T01:01:58.4356081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4356159Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4356399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4356469Z return func(*args, **kwargs) 2025-10-10T01:01:58.4356720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4356844Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4357139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4357225Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4357514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4357660Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4357921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4358000Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4358004Z 2025-10-10T01:01:58.4358114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4358306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4358410Z res = mod(**inputs) 2025-10-10T01:01:58.4358660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4358730Z outputs = self.roberta( 2025-10-10T01:01:58.4358985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4359057Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4359313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4359383Z layer_outputs = layer_module( 2025-10-10T01:01:58.4359595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4359682Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4359915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4359998Z return func(*args, **kwargs) 2025-10-10T01:01:58.4360250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4360334Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4360581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4360652Z return func(*args, **kwargs) 2025-10-10T01:01:58.4360925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4360994Z self_outputs = self.self( 2025-10-10T01:01:58.4361232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4361299Z return func(*args, **kwargs) 2025-10-10T01:01:58.4361551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4361762Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4361765Z 2025-10-10T01:01:58.4361866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4362068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4362133Z res = mod(**inputs) 2025-10-10T01:01:58.4362388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4362466Z outputs = self.roberta( 2025-10-10T01:01:58.4362725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4362804Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4363129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4363209Z layer_outputs = layer_module( 2025-10-10T01:01:58.4363421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4363499Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4363738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4363810Z return func(*args, **kwargs) 2025-10-10T01:01:58.4364079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4364165Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4364408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4364526Z return func(*args, **kwargs) 2025-10-10T01:01:58.4364791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4364872Z self_outputs = self.self( 2025-10-10T01:01:58.4365118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4365191Z return func(*args, **kwargs) 2025-10-10T01:01:58.4365460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4365534Z self.key(current_states) 2025-10-10T01:01:58.4365537Z 2025-10-10T01:01:58.4365663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4365852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4365926Z res = mod(**inputs) 2025-10-10T01:01:58.4366179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4366246Z outputs = self.roberta( 2025-10-10T01:01:58.4366501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4366573Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4366829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4366901Z layer_outputs = layer_module( 2025-10-10T01:01:58.4367117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4367205Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4367444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4367525Z return func(*args, **kwargs) 2025-10-10T01:01:58.4367781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4367869Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4368109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4368176Z return func(*args, **kwargs) 2025-10-10T01:01:58.4368439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4368510Z self_outputs = self.self( 2025-10-10T01:01:58.4368759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4368825Z return func(*args, **kwargs) 2025-10-10T01:01:58.4369073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4369198Z self.value(current_states) 2025-10-10T01:01:58.4369202Z 2025-10-10T01:01:58.4369285Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4369395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4369584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4369647Z res = mod(**inputs) 2025-10-10T01:01:58.4369932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4370002Z outputs = self.roberta( 2025-10-10T01:01:58.4370262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4370335Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4370622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4370716Z layer_outputs = layer_module( 2025-10-10T01:01:58.4370944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4371033Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4371289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4371364Z return func(*args, **kwargs) 2025-10-10T01:01:58.4371622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4371706Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4371959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4372029Z return func(*args, **kwargs) 2025-10-10T01:01:58.4372293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4372363Z self_outputs = self.self( 2025-10-10T01:01:58.4372604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4372682Z return func(*args, **kwargs) 2025-10-10T01:01:58.4372972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4373116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4373120Z 2025-10-10T01:01:58.4373228Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4373435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4373502Z res = mod(**inputs) 2025-10-10T01:01:58.4373774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4373853Z outputs = self.roberta( 2025-10-10T01:01:58.4374118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4374204Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4374471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4374546Z layer_outputs = layer_module( 2025-10-10T01:01:58.4374782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4374864Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4375121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4375214Z return func(*args, **kwargs) 2025-10-10T01:01:58.4375497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4375592Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4375837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4375917Z return func(*args, **kwargs) 2025-10-10T01:01:58.4376181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4376320Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4376584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4376668Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4376690Z 2025-10-10T01:01:58.4376832Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4377035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4377108Z res = mod(**inputs) 2025-10-10T01:01:58.4377371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4377442Z outputs = self.roberta( 2025-10-10T01:01:58.4377712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4377788Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4378056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4378129Z layer_outputs = layer_module( 2025-10-10T01:01:58.4378363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4378449Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4378699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4378782Z return func(*args, **kwargs) 2025-10-10T01:01:58.4379061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4379161Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4379439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4379518Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4379825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4379955Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4380230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4380316Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4380319Z 2025-10-10T01:01:58.4380433Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4380635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4380702Z res = mod(**inputs) 2025-10-10T01:01:58.4380972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4381043Z outputs = self.roberta( 2025-10-10T01:01:58.4381311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4381386Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4381682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4381766Z layer_outputs = layer_module( 2025-10-10T01:01:58.4381988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4382075Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4382328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4382402Z return func(*args, **kwargs) 2025-10-10T01:01:58.4382684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4382773Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4383058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4383178Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4383505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4383635Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4383914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4384046Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4384278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4384362Z return self.act(input) 2025-10-10T01:01:58.4384366Z 2025-10-10T01:01:58.4384477Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4384689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4384772Z res = mod(**inputs) 2025-10-10T01:01:58.4385056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4385138Z outputs = self.roberta( 2025-10-10T01:01:58.4385418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4385501Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4385781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4385859Z layer_outputs = layer_module( 2025-10-10T01:01:58.4386107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4386196Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4386579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4386669Z return func(*args, **kwargs) 2025-10-10T01:01:58.4386957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4387061Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4387343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4387439Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4387758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4387903Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4388165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4388276Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4388298Z 2025-10-10T01:01:58.4388416Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4388615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4388692Z res = mod(**inputs) 2025-10-10T01:01:58.4388952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4389024Z outputs = self.roberta( 2025-10-10T01:01:58.4389296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4389371Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4389641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4389716Z layer_outputs = layer_module( 2025-10-10T01:01:58.4389983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4390072Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4390310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4390385Z return func(*args, **kwargs) 2025-10-10T01:01:58.4390637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4390726Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4390958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4391025Z return func(*args, **kwargs) 2025-10-10T01:01:58.4391286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4391359Z self_outputs = self.self( 2025-10-10T01:01:58.4391601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4391668Z return func(*args, **kwargs) 2025-10-10T01:01:58.4391918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4392130Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4392134Z 2025-10-10T01:01:58.4392234Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4392432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4392497Z res = mod(**inputs) 2025-10-10T01:01:58.4392760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4392832Z outputs = self.roberta( 2025-10-10T01:01:58.4393081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4396424Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4396715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4396792Z layer_outputs = layer_module( 2025-10-10T01:01:58.4397034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4397126Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4397403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4397487Z return func(*args, **kwargs) 2025-10-10T01:01:58.4397818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4397927Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4398194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4398324Z return func(*args, **kwargs) 2025-10-10T01:01:58.4398616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4398693Z self_outputs = self.self( 2025-10-10T01:01:58.4398939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4399019Z return func(*args, **kwargs) 2025-10-10T01:01:58.4399286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4399410Z self.key(current_states) 2025-10-10T01:01:58.4399415Z 2025-10-10T01:01:58.4399536Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4399758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4399830Z res = mod(**inputs) 2025-10-10T01:01:58.4400099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4400180Z outputs = self.roberta( 2025-10-10T01:01:58.4400445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4400530Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4400794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4400869Z layer_outputs = layer_module( 2025-10-10T01:01:58.4401116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4401199Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4401445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4401518Z return func(*args, **kwargs) 2025-10-10T01:01:58.4401783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4401878Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4402124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4402202Z return func(*args, **kwargs) 2025-10-10T01:01:58.4402467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4402550Z self_outputs = self.self( 2025-10-10T01:01:58.4402799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4402870Z return func(*args, **kwargs) 2025-10-10T01:01:58.4403199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4403276Z self.value(current_states) 2025-10-10T01:01:58.4403280Z 2025-10-10T01:01:58.4403377Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4403496Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4403704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4403778Z res = mod(**inputs) 2025-10-10T01:01:58.4404035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4404116Z outputs = self.roberta( 2025-10-10T01:01:58.4404388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4404464Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4404733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4404807Z layer_outputs = layer_module( 2025-10-10T01:01:58.4405040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4405121Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4405387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4405456Z return func(*args, **kwargs) 2025-10-10T01:01:58.4405745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4405853Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4406093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4406168Z return func(*args, **kwargs) 2025-10-10T01:01:58.4406425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4406499Z self_outputs = self.self( 2025-10-10T01:01:58.4406752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4406823Z return func(*args, **kwargs) 2025-10-10T01:01:58.4407103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4407238Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4407244Z 2025-10-10T01:01:58.4407360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4407560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4407626Z res = mod(**inputs) 2025-10-10T01:01:58.4407891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4407961Z outputs = self.roberta( 2025-10-10T01:01:58.4408227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4408301Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4408564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4408645Z layer_outputs = layer_module( 2025-10-10T01:01:58.4408872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4408965Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4409212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4409317Z return func(*args, **kwargs) 2025-10-10T01:01:58.4409590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4409676Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4409931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4410001Z return func(*args, **kwargs) 2025-10-10T01:01:58.4410284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4410431Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4410732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4410835Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4410841Z 2025-10-10T01:01:58.4410956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4411177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4411249Z res = mod(**inputs) 2025-10-10T01:01:58.4411525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4411609Z outputs = self.roberta( 2025-10-10T01:01:58.4411888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4411974Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4412291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4412377Z layer_outputs = layer_module( 2025-10-10T01:01:58.4412620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4412709Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4412981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4413056Z return func(*args, **kwargs) 2025-10-10T01:01:58.4413345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4413439Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4413724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4413822Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4414149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4414292Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4414577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4414674Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4414678Z 2025-10-10T01:01:58.4414791Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4415008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4415088Z res = mod(**inputs) 2025-10-10T01:01:58.4415375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4415461Z outputs = self.roberta( 2025-10-10T01:01:58.4415745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4415825Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4416140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4416218Z layer_outputs = layer_module( 2025-10-10T01:01:58.4416468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4416555Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4416826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4416910Z return func(*args, **kwargs) 2025-10-10T01:01:58.4417212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4417310Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4417571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4417662Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4417964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4418089Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4418363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4418479Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4418723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4418815Z return self.act(input) 2025-10-10T01:01:58.4418819Z 2025-10-10T01:01:58.4418928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4419141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4419210Z res = mod(**inputs) 2025-10-10T01:01:58.4419499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4419573Z outputs = self.roberta( 2025-10-10T01:01:58.4419869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4419947Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4420234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4420323Z layer_outputs = layer_module( 2025-10-10T01:01:58.4420570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4420661Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4420930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4421008Z return func(*args, **kwargs) 2025-10-10T01:01:58.4421303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4421392Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4421683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4421765Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4422093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4422251Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4422540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4422660Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4422664Z 2025-10-10T01:01:58.4422776Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4422998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4423068Z res = mod(**inputs) 2025-10-10T01:01:58.4423345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4423428Z outputs = self.roberta( 2025-10-10T01:01:58.4423705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4423811Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4424097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4424178Z layer_outputs = layer_module( 2025-10-10T01:01:58.4424432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4424520Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4424796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4424874Z return func(*args, **kwargs) 2025-10-10T01:01:58.4425168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4425261Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4425568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4425655Z return func(*args, **kwargs) 2025-10-10T01:01:58.4425951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4426046Z self_outputs = self.self( 2025-10-10T01:01:58.4426307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4426603Z return func(*args, **kwargs) 2025-10-10T01:01:58.4426897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4427141Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4427145Z 2025-10-10T01:01:58.4427265Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4427492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4427564Z res = mod(**inputs) 2025-10-10T01:01:58.4427848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4427936Z outputs = self.roberta( 2025-10-10T01:01:58.4428219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4428308Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4428588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4428666Z layer_outputs = layer_module( 2025-10-10T01:01:58.4428918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4429010Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4429284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4429361Z return func(*args, **kwargs) 2025-10-10T01:01:58.4429669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4429767Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4430029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4430112Z return func(*args, **kwargs) 2025-10-10T01:01:58.4430394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4430480Z self_outputs = self.self( 2025-10-10T01:01:58.4430743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4430847Z return func(*args, **kwargs) 2025-10-10T01:01:58.4431145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4431225Z self.key(current_states) 2025-10-10T01:01:58.4431229Z 2025-10-10T01:01:58.4431349Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4431719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4431795Z res = mod(**inputs) 2025-10-10T01:01:58.4432093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4432169Z outputs = self.roberta( 2025-10-10T01:01:58.4432463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4432616Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4432898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4432985Z layer_outputs = layer_module( 2025-10-10T01:01:58.4433228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4433321Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4433582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4433663Z return func(*args, **kwargs) 2025-10-10T01:01:58.4433942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4434031Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4434300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4434377Z return func(*args, **kwargs) 2025-10-10T01:01:58.4434663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4434743Z self_outputs = self.self( 2025-10-10T01:01:58.4435004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4435090Z return func(*args, **kwargs) 2025-10-10T01:01:58.4435369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4435455Z self.value(current_states) 2025-10-10T01:01:58.4435460Z 2025-10-10T01:01:58.4435550Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4435663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4435889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4435952Z res = mod(**inputs) 2025-10-10T01:01:58.4436206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4436305Z outputs = self.roberta( 2025-10-10T01:01:58.4436568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4436640Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4436898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4436977Z layer_outputs = layer_module( 2025-10-10T01:01:58.4437200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4437288Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4437556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4437630Z return func(*args, **kwargs) 2025-10-10T01:01:58.4437893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4437979Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4438226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4438295Z return func(*args, **kwargs) 2025-10-10T01:01:58.4438551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4438630Z self_outputs = self.self( 2025-10-10T01:01:58.4438894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4438989Z return func(*args, **kwargs) 2025-10-10T01:01:58.4439243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4439384Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4439389Z 2025-10-10T01:01:58.4439492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4439684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4439756Z res = mod(**inputs) 2025-10-10T01:01:58.4440011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4440088Z outputs = self.roberta( 2025-10-10T01:01:58.4440345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4440422Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4440688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4440761Z layer_outputs = layer_module( 2025-10-10T01:01:58.4440991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4441071Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4441316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4441384Z return func(*args, **kwargs) 2025-10-10T01:01:58.4441639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4441727Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4441966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4442045Z return func(*args, **kwargs) 2025-10-10T01:01:58.4442300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4442456Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4442723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4442807Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4442810Z 2025-10-10T01:01:58.4442920Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4443117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4443190Z res = mod(**inputs) 2025-10-10T01:01:58.4443448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4443537Z outputs = self.roberta( 2025-10-10T01:01:58.4443806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4443885Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4444174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4444251Z layer_outputs = layer_module( 2025-10-10T01:01:58.4444499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4444587Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4444834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4444912Z return func(*args, **kwargs) 2025-10-10T01:01:58.4445212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4445307Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4445590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4445677Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4446008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4446132Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4446403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4446490Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4446493Z 2025-10-10T01:01:58.4446600Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4446815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4446882Z res = mod(**inputs) 2025-10-10T01:01:58.4447152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4447223Z outputs = self.roberta( 2025-10-10T01:01:58.4447488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4447570Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4447834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4447912Z layer_outputs = layer_module( 2025-10-10T01:01:58.4448136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4448225Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4448472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4448544Z return func(*args, **kwargs) 2025-10-10T01:01:58.4448850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4448940Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4449221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4449312Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4449609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4449739Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4450020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4450145Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4450362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4450444Z return self.act(input) 2025-10-10T01:01:58.4450448Z 2025-10-10T01:01:58.4450551Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4450754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4450829Z res = mod(**inputs) 2025-10-10T01:01:58.4451098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4451177Z outputs = self.roberta( 2025-10-10T01:01:58.4451469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4451566Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4451836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4451911Z layer_outputs = layer_module( 2025-10-10T01:01:58.4452141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4452223Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4452498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4452580Z return func(*args, **kwargs) 2025-10-10T01:01:58.4452859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4452958Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4453242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4453329Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4453631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4453771Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4454044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4454129Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4454133Z 2025-10-10T01:01:58.4454247Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4454449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4454518Z res = mod(**inputs) 2025-10-10T01:01:58.4454792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4454867Z outputs = self.roberta( 2025-10-10T01:01:58.4455152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4455263Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4455557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4455636Z layer_outputs = layer_module( 2025-10-10T01:01:58.4455879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4455974Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4456254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4456358Z return func(*args, **kwargs) 2025-10-10T01:01:58.4456637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4456728Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4457002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4457075Z return func(*args, **kwargs) 2025-10-10T01:01:58.4457347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4457421Z self_outputs = self.self( 2025-10-10T01:01:58.4457669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4457747Z return func(*args, **kwargs) 2025-10-10T01:01:58.4458045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4458278Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4458283Z 2025-10-10T01:01:58.4458397Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4458618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4458689Z res = mod(**inputs) 2025-10-10T01:01:58.4458971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4459054Z outputs = self.roberta( 2025-10-10T01:01:58.4459339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4459424Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4459716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4459796Z layer_outputs = layer_module( 2025-10-10T01:01:58.4460047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4460135Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4460411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4460488Z return func(*args, **kwargs) 2025-10-10T01:01:58.4460789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4460883Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4461155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4461246Z return func(*args, **kwargs) 2025-10-10T01:01:58.4461544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4461631Z self_outputs = self.self( 2025-10-10T01:01:58.4461939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4462016Z return func(*args, **kwargs) 2025-10-10T01:01:58.4462309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4462390Z self.key(current_states) 2025-10-10T01:01:58.4462394Z 2025-10-10T01:01:58.4462516Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4462734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4462808Z res = mod(**inputs) 2025-10-10T01:01:58.4463123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4463202Z outputs = self.roberta( 2025-10-10T01:01:58.4463495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4463578Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4463871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4463950Z layer_outputs = layer_module( 2025-10-10T01:01:58.4464195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4464290Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4464559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4464681Z return func(*args, **kwargs) 2025-10-10T01:01:58.4464973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4465065Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4465344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4465424Z return func(*args, **kwargs) 2025-10-10T01:01:58.4465720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4465800Z self_outputs = self.self( 2025-10-10T01:01:58.4466071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4466148Z return func(*args, **kwargs) 2025-10-10T01:01:58.4466533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4466637Z self.value(current_states) 2025-10-10T01:01:58.4466641Z 2025-10-10T01:01:58.4466736Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4466861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4467083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4467158Z res = mod(**inputs) 2025-10-10T01:01:58.4467453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4467531Z outputs = self.roberta( 2025-10-10T01:01:58.4467824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4467899Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4468164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4468250Z layer_outputs = layer_module( 2025-10-10T01:01:58.4468476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4468590Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4468838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4468919Z return func(*args, **kwargs) 2025-10-10T01:01:58.4469181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4469266Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4469517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4469588Z return func(*args, **kwargs) 2025-10-10T01:01:58.4469898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4469979Z self_outputs = self.self( 2025-10-10T01:01:58.4470274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4470359Z return func(*args, **kwargs) 2025-10-10T01:01:58.4470642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4470798Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4470802Z 2025-10-10T01:01:58.4470914Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4471130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4471210Z res = mod(**inputs) 2025-10-10T01:01:58.4471513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4471614Z outputs = self.roberta( 2025-10-10T01:01:58.4471895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4471983Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4472273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4472346Z layer_outputs = layer_module( 2025-10-10T01:01:58.4472578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4472658Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4472932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4473010Z return func(*args, **kwargs) 2025-10-10T01:01:58.4473290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4473388Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4473648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4473732Z return func(*args, **kwargs) 2025-10-10T01:01:58.4474009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4474158Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4474437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4474527Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4474531Z 2025-10-10T01:01:58.4474655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4474868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4474947Z res = mod(**inputs) 2025-10-10T01:01:58.4475226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4475322Z outputs = self.roberta( 2025-10-10T01:01:58.4475610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4475688Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4475975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4476052Z layer_outputs = layer_module( 2025-10-10T01:01:58.4476292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4476401Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4476673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4476756Z return func(*args, **kwargs) 2025-10-10T01:01:58.4477041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4477140Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4477419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4477503Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4477828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4477960Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4478301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4478391Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4478395Z 2025-10-10T01:01:58.4478509Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4478737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4478810Z res = mod(**inputs) 2025-10-10T01:01:58.4479093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4479168Z outputs = self.roberta( 2025-10-10T01:01:58.4479451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4479530Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4479810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4479894Z layer_outputs = layer_module( 2025-10-10T01:01:58.4480132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4480224Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4480482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4480557Z return func(*args, **kwargs) 2025-10-10T01:01:58.4480839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4480930Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4481213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4481299Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4481622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4481752Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4482049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4482178Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4482413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4482499Z return self.act(input) 2025-10-10T01:01:58.4482503Z 2025-10-10T01:01:58.4482617Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4482835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4482919Z res = mod(**inputs) 2025-10-10T01:01:58.4483233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4483315Z outputs = self.roberta( 2025-10-10T01:01:58.4483592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4483670Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4483955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4484033Z layer_outputs = layer_module( 2025-10-10T01:01:58.4484277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4484363Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4484646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4484741Z return func(*args, **kwargs) 2025-10-10T01:01:58.4485020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4485122Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4485398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4485487Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4485804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4485947Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4486237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4486333Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4486339Z 2025-10-10T01:01:58.4486461Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4486681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4486764Z res = mod(**inputs) 2025-10-10T01:01:58.4487063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4487139Z outputs = self.roberta( 2025-10-10T01:01:58.4487427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4487507Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4487793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4487873Z layer_outputs = layer_module( 2025-10-10T01:01:58.4488118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4488212Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4488474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4488578Z return func(*args, **kwargs) 2025-10-10T01:01:58.4488856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4488946Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4489215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4489291Z return func(*args, **kwargs) 2025-10-10T01:01:58.4489579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4489661Z self_outputs = self.self( 2025-10-10T01:01:58.4489945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4490022Z return func(*args, **kwargs) 2025-10-10T01:01:58.4490311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4490544Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4490548Z 2025-10-10T01:01:58.4490660Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4490880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4490950Z res = mod(**inputs) 2025-10-10T01:01:58.4491248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4491347Z outputs = self.roberta( 2025-10-10T01:01:58.4491627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4491713Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4491995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4492080Z layer_outputs = layer_module( 2025-10-10T01:01:58.4492319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4492405Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4492682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4492760Z return func(*args, **kwargs) 2025-10-10T01:01:58.4493061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4493155Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4493423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4493510Z return func(*args, **kwargs) 2025-10-10T01:01:58.4493799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4493884Z self_outputs = self.self( 2025-10-10T01:01:58.4494152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4494236Z return func(*args, **kwargs) 2025-10-10T01:01:58.4494524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4494607Z self.key(current_states) 2025-10-10T01:01:58.4494612Z 2025-10-10T01:01:58.4494739Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4494961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4495063Z res = mod(**inputs) 2025-10-10T01:01:58.4495350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4495428Z outputs = self.roberta( 2025-10-10T01:01:58.4495724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4495805Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4496099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4496179Z layer_outputs = layer_module( 2025-10-10T01:01:58.4496446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4496543Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4496811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4496896Z return func(*args, **kwargs) 2025-10-10T01:01:58.4497185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4497282Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4497549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4497626Z return func(*args, **kwargs) 2025-10-10T01:01:58.4497920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4498036Z self_outputs = self.self( 2025-10-10T01:01:58.4498313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4498389Z return func(*args, **kwargs) 2025-10-10T01:01:58.4498675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4498763Z self.value(current_states) 2025-10-10T01:01:58.4498767Z 2025-10-10T01:01:58.4498859Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4498984Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4499204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4499276Z res = mod(**inputs) 2025-10-10T01:01:58.4499571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4499651Z outputs = self.roberta( 2025-10-10T01:01:58.4499945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4500027Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4500320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4500401Z layer_outputs = layer_module( 2025-10-10T01:01:58.4500645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4500740Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4501006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4501091Z return func(*args, **kwargs) 2025-10-10T01:01:58.4501377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4501472Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4501746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4501846Z return func(*args, **kwargs) 2025-10-10T01:01:58.4502150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4502230Z self_outputs = self.self( 2025-10-10T01:01:58.4502505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4502591Z return func(*args, **kwargs) 2025-10-10T01:01:58.4502887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4503042Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4503049Z 2025-10-10T01:01:58.4503184Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4503415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4503490Z res = mod(**inputs) 2025-10-10T01:01:58.4503782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4503872Z outputs = self.roberta( 2025-10-10T01:01:58.4504162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4504251Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4504539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4504620Z layer_outputs = layer_module( 2025-10-10T01:01:58.4504901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4505010Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4505283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4505363Z return func(*args, **kwargs) 2025-10-10T01:01:58.4505660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4505753Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4506020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4506105Z return func(*args, **kwargs) 2025-10-10T01:01:58.4506480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4506646Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4506943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4507039Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4507044Z 2025-10-10T01:01:58.4507174Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4507393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4507477Z res = mod(**inputs) 2025-10-10T01:01:58.4507770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4507848Z outputs = self.roberta( 2025-10-10T01:01:58.4508150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4508231Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4508537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4508617Z layer_outputs = layer_module( 2025-10-10T01:01:58.4508875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4509015Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4509283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4509369Z return func(*args, **kwargs) 2025-10-10T01:01:58.4509658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4509761Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4510054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4510165Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4510506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4510643Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4510938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4511031Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4511035Z 2025-10-10T01:01:58.4511159Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4511377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4511450Z res = mod(**inputs) 2025-10-10T01:01:58.4511764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4511861Z outputs = self.roberta( 2025-10-10T01:01:58.4512157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4512238Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4512523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4512612Z layer_outputs = layer_module( 2025-10-10T01:01:58.4512858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4512952Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4513229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4513312Z return func(*args, **kwargs) 2025-10-10T01:01:58.4513600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4513695Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4513987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4514076Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4514408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4514542Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4514826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4514961Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4515201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4515290Z return self.act(input) 2025-10-10T01:01:58.4515296Z 2025-10-10T01:01:58.4515411Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4515638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4515734Z res = mod(**inputs) 2025-10-10T01:01:58.4516025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4516110Z outputs = self.roberta( 2025-10-10T01:01:58.4516401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4516490Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4516784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4516868Z layer_outputs = layer_module( 2025-10-10T01:01:58.4517149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4517239Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4517522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4517604Z return func(*args, **kwargs) 2025-10-10T01:01:58.4517892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4517994Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4518278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4518371Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4518716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4518892Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4519188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4519282Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4519286Z 2025-10-10T01:01:58.4519412Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4519634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4519715Z res = mod(**inputs) 2025-10-10T01:01:58.4520003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4520081Z outputs = self.roberta( 2025-10-10T01:01:58.4520379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4520465Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4520764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4520848Z layer_outputs = layer_module( 2025-10-10T01:01:58.4521103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4521191Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4521463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4521550Z return func(*args, **kwargs) 2025-10-10T01:01:58.4521841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4521941Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4522218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4522295Z return func(*args, **kwargs) 2025-10-10T01:01:58.4522595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4522695Z self_outputs = self.self( 2025-10-10T01:01:58.4522991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4523069Z return func(*args, **kwargs) 2025-10-10T01:01:58.4523359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-10-10T01:01:58.4523601Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-10-10T01:01:58.4523605Z 2025-10-10T01:01:58.4523723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4523972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4524047Z res = mod(**inputs) 2025-10-10T01:01:58.4524343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4524423Z outputs = self.roberta( 2025-10-10T01:01:58.4524709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4524799Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4525089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4525177Z layer_outputs = layer_module( 2025-10-10T01:01:58.4525423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4525548Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4525826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4525905Z return func(*args, **kwargs) 2025-10-10T01:01:58.4526202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4526294Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4526569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4526645Z return func(*args, **kwargs) 2025-10-10T01:01:58.4526935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4527023Z self_outputs = self.self( 2025-10-10T01:01:58.4527294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4527382Z return func(*args, **kwargs) 2025-10-10T01:01:58.4527672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-10-10T01:01:58.4527752Z self.key(current_states) 2025-10-10T01:01:58.4527756Z 2025-10-10T01:01:58.4527881Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4528102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4528182Z res = mod(**inputs) 2025-10-10T01:01:58.4528471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4528549Z outputs = self.roberta( 2025-10-10T01:01:58.4528845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4528929Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4529227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4529308Z layer_outputs = layer_module( 2025-10-10T01:01:58.4529589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4529676Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4529944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4530028Z return func(*args, **kwargs) 2025-10-10T01:01:58.4530315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4530412Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4530677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4530787Z return func(*args, **kwargs) 2025-10-10T01:01:58.4531091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4531171Z self_outputs = self.self( 2025-10-10T01:01:58.4531599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4531689Z return func(*args, **kwargs) 2025-10-10T01:01:58.4531983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-10-10T01:01:58.4532074Z self.value(current_states) 2025-10-10T01:01:58.4532078Z 2025-10-10T01:01:58.4532172Z cudagraph partition due to non gpu ops 2025-10-10T01:01:58.4532300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4532592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4532676Z res = mod(**inputs) 2025-10-10T01:01:58.4532970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4533049Z outputs = self.roberta( 2025-10-10T01:01:58.4533343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4533423Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4533720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4533800Z layer_outputs = layer_module( 2025-10-10T01:01:58.4534049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4534147Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4534442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4534528Z return func(*args, **kwargs) 2025-10-10T01:01:58.4534820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4534916Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4535201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4535279Z return func(*args, **kwargs) 2025-10-10T01:01:58.4535576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-10-10T01:01:58.4535655Z self_outputs = self.self( 2025-10-10T01:01:58.4535988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4536070Z return func(*args, **kwargs) 2025-10-10T01:01:58.4536361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-10-10T01:01:58.4536522Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-10-10T01:01:58.4536557Z 2025-10-10T01:01:58.4536674Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4536901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4536976Z res = mod(**inputs) 2025-10-10T01:01:58.4537264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4537349Z outputs = self.roberta( 2025-10-10T01:01:58.4537636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4537728Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4538041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4538130Z layer_outputs = layer_module( 2025-10-10T01:01:58.4538393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4538478Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4538751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4538827Z return func(*args, **kwargs) 2025-10-10T01:01:58.4539119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-10-10T01:01:58.4539209Z self_attention_outputs = self.attention( 2025-10-10T01:01:58.4539497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4539602Z return func(*args, **kwargs) 2025-10-10T01:01:58.4539879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-10-10T01:01:58.4540029Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:01:58.4540307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-10-10T01:01:58.4540400Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4540410Z 2025-10-10T01:01:58.4540522Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4540732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4540810Z res = mod(**inputs) 2025-10-10T01:01:58.4541093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4541181Z outputs = self.roberta( 2025-10-10T01:01:58.4541469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4541551Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4541846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4541926Z layer_outputs = layer_module( 2025-10-10T01:01:58.4542177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4542264Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4542531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4542617Z return func(*args, **kwargs) 2025-10-10T01:01:58.4542911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4543013Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4543301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4543420Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4543753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4543888Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4544187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-10-10T01:01:58.4544280Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4544284Z 2025-10-10T01:01:58.4544408Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4544652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4544726Z res = mod(**inputs) 2025-10-10T01:01:58.4545023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4545101Z outputs = self.roberta( 2025-10-10T01:01:58.4545397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4545478Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4545770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4545850Z layer_outputs = layer_module( 2025-10-10T01:01:58.4546095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4546226Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4546571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4546663Z return func(*args, **kwargs) 2025-10-10T01:01:58.4546953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4547047Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4547340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4547426Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4547759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-10-10T01:01:58.4547893Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:01:58.4548198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-10-10T01:01:58.4548322Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:01:58.4548557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:01:58.4548646Z return self.act(input) 2025-10-10T01:01:58.4548650Z 2025-10-10T01:01:58.4548763Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4548984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4549056Z res = mod(**inputs) 2025-10-10T01:01:58.4549336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-10-10T01:01:58.4549420Z outputs = self.roberta( 2025-10-10T01:01:58.4549700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-10-10T01:01:58.4549793Z encoder_outputs = self.encoder( 2025-10-10T01:01:58.4550073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-10-10T01:01:58.4550175Z layer_outputs = layer_module( 2025-10-10T01:01:58.4550424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:01:58.4550510Z return super().__call__(*args, **kwargs) 2025-10-10T01:01:58.4550784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:01:58.4550859Z return func(*args, **kwargs) 2025-10-10T01:01:58.4551148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-10-10T01:01:58.4551241Z layer_output = apply_chunking_to_forward( 2025-10-10T01:01:58.4551540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:01:58.4551635Z return forward_fn(*input_tensors) 2025-10-10T01:01:58.4551954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-10-10T01:01:58.4552109Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:01:58.4552393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-10-10T01:01:58.4552483Z hidden_states = self.dense(hidden_states) 2025-10-10T01:01:58.4552494Z 2025-10-10T01:01:58.4552607Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4552825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4552923Z res = mod(**inputs) 2025-10-10T01:01:58.4553224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 987, in forward 2025-10-10T01:01:58.4553341Z prediction_scores = self.lm_head(sequence_output) 2025-10-10T01:01:58.4553630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1120, in forward 2025-10-10T01:01:58.4553710Z x = self.dense(features) 2025-10-10T01:01:58.4553714Z 2025-10-10T01:01:58.4553833Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4554056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4554130Z res = mod(**inputs) 2025-10-10T01:01:58.4554395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 987, in forward 2025-10-10T01:01:58.4554495Z prediction_scores = self.lm_head(sequence_output) 2025-10-10T01:01:58.4554779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1125, in forward 2025-10-10T01:01:58.4554850Z x = self.decoder(x) 2025-10-10T01:01:58.4554854Z 2025-10-10T01:01:58.4554966Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:01:58.4555168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:01:58.4555242Z res = mod(**inputs) 2025-10-10T01:01:58.4555518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 993, in forward 2025-10-10T01:01:58.4555591Z lm_loss = self.loss_function( 2025-10-10T01:01:58.4555842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T01:01:58.4556015Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T01:01:58.4556278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T01:01:58.4556471Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T01:01:58.4556504Z 2025-10-10T01:02:09.0824140Z Compilation time (from dynamo_timed): 17.345757028 2025-10-10T01:02:09.0939417Z pass 2025-10-10T01:02:09.0940494Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:09.0941402Z TIMING: _recursive_pre_grad_passes:0.00743 _recursive_joint_graph_passes:0.40099 _recursive_post_grad_passes:0.07596 async_compile.wait:0.75621 code_gen:9.85303 inductor_compile:11.22069 backend_compile:14.47999 gc:0.00046 entire_frame_compile:17.34576 total_wall_time:17.34576 2025-10-10T01:02:09.0942674Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:7334 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:1966 2025-10-10T01:02:09.0943583Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-10-10T01:02:11.9289182Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:02:11.9291798Z import pynvml # type: ignore[import] 2025-10-10T01:02:15.4205470Z 2025-10-10T01:02:16.4170848Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:02:16.4171316Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:02:16.4181623Z cpu eval T5ForConditionalGeneration 2025-10-10T01:02:17.5078775Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:17.9294911Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:18.3640678Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:27.6560685Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6561222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6561662Z res = mod(**inputs) 2025-10-10T01:02:27.6562089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6562532Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6562980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6563403Z layer_outputs = layer_module( 2025-10-10T01:02:27.6563793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6564196Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6564647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6565073Z return func(*args, **kwargs) 2025-10-10T01:02:27.6565477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6565900Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6566321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6566743Z return func(*args, **kwargs) 2025-10-10T01:02:27.6567125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6567555Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6567992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6568408Z return func(*args, **kwargs) 2025-10-10T01:02:27.6568800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 548, in forward 2025-10-10T01:02:27.6569235Z position_bias = position_bias + causal_mask 2025-10-10T01:02:27.6569718Z 2025-10-10T01:02:27.6569840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6570235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6570603Z res = mod(**inputs) 2025-10-10T01:02:27.6571012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6571428Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6571840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6572252Z layer_outputs = layer_module( 2025-10-10T01:02:27.6572698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6573116Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6573537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6573955Z return func(*args, **kwargs) 2025-10-10T01:02:27.6574349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6574760Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6575185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6575593Z return func(*args, **kwargs) 2025-10-10T01:02:27.6576007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6576537Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6576950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6577360Z return func(*args, **kwargs) 2025-10-10T01:02:27.6577748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6578155Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6578305Z 2025-10-10T01:02:27.6578432Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6578821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6579182Z res = mod(**inputs) 2025-10-10T01:02:27.6579566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6579993Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6580401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6580821Z layer_outputs = layer_module( 2025-10-10T01:02:27.6581215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6581660Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6582092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6582514Z return func(*args, **kwargs) 2025-10-10T01:02:27.6582911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6583339Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6583776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6584198Z return func(*args, **kwargs) 2025-10-10T01:02:27.6584602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6585021Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6585457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6585908Z return func(*args, **kwargs) 2025-10-10T01:02:27.6586311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6587076Z key_states = self.k(current_states) 2025-10-10T01:02:27.6587238Z 2025-10-10T01:02:27.6587360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6587784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6588166Z res = mod(**inputs) 2025-10-10T01:02:27.6588530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6588942Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6589331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6589738Z layer_outputs = layer_module( 2025-10-10T01:02:27.6590129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6590809Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6591229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6591632Z return func(*args, **kwargs) 2025-10-10T01:02:27.6592026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6592444Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6592908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6593319Z return func(*args, **kwargs) 2025-10-10T01:02:27.6593709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6594141Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6594568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6594999Z return func(*args, **kwargs) 2025-10-10T01:02:27.6595400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6595805Z value_states = self.v(current_states) 2025-10-10T01:02:27.6595962Z 2025-10-10T01:02:27.6596055Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6596294Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6596560Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6596951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6597304Z res = mod(**inputs) 2025-10-10T01:02:27.6597686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6598091Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6598496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6598893Z layer_outputs = layer_module( 2025-10-10T01:02:27.6599274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6599671Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6600093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6600500Z return func(*args, **kwargs) 2025-10-10T01:02:27.6600887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6601315Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6601729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6602135Z return func(*args, **kwargs) 2025-10-10T01:02:27.6602525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6602948Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6603374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6603788Z return func(*args, **kwargs) 2025-10-10T01:02:27.6604202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6604620Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6604774Z 2025-10-10T01:02:27.6604893Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6605300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6605661Z res = mod(**inputs) 2025-10-10T01:02:27.6606048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6606459Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6606861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6607264Z layer_outputs = layer_module( 2025-10-10T01:02:27.6607662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6608067Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6608483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6608883Z return func(*args, **kwargs) 2025-10-10T01:02:27.6609264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6609690Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6610105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6610554Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6611000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6611409Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6611565Z 2025-10-10T01:02:27.6611688Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6612089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6612450Z res = mod(**inputs) 2025-10-10T01:02:27.6612839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6613262Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6613652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6614067Z layer_outputs = layer_module( 2025-10-10T01:02:27.6614454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6614862Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6615280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6615707Z return func(*args, **kwargs) 2025-10-10T01:02:27.6616114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6616608Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6617046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6617509Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6617987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6618410Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6618561Z 2025-10-10T01:02:27.6618686Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6619087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6619446Z res = mod(**inputs) 2025-10-10T01:02:27.6619905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6620331Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6620744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6621159Z layer_outputs = layer_module( 2025-10-10T01:02:27.6621547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6621952Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6622388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6622814Z return func(*args, **kwargs) 2025-10-10T01:02:27.6623232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6623699Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6624144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6624619Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6625091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6625517Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6625726Z 2025-10-10T01:02:27.6625882Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6626283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6626747Z res = mod(**inputs) 2025-10-10T01:02:27.6627136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6627558Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6627970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6628392Z layer_outputs = layer_module( 2025-10-10T01:02:27.6628784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6629187Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6629615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6630048Z return func(*args, **kwargs) 2025-10-10T01:02:27.6630449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6630881Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6631308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6632131Z return func(*args, **kwargs) 2025-10-10T01:02:27.6632537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6633024Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6633449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6633888Z return func(*args, **kwargs) 2025-10-10T01:02:27.6634285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6634717Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6634872Z 2025-10-10T01:02:27.6634999Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6635400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6635797Z res = mod(**inputs) 2025-10-10T01:02:27.6636178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6636577Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6636964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6637364Z layer_outputs = layer_module( 2025-10-10T01:02:27.6637742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6638133Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6638546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6638941Z return func(*args, **kwargs) 2025-10-10T01:02:27.6639350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6639788Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6640204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6640614Z return func(*args, **kwargs) 2025-10-10T01:02:27.6640997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6641411Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6641829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6642231Z return func(*args, **kwargs) 2025-10-10T01:02:27.6642618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6643043Z key_states = self.k(current_states) 2025-10-10T01:02:27.6643206Z 2025-10-10T01:02:27.6643327Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6643732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6644096Z res = mod(**inputs) 2025-10-10T01:02:27.6644491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6644893Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6645291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6645696Z layer_outputs = layer_module( 2025-10-10T01:02:27.6646069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6646487Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6646902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6647316Z return func(*args, **kwargs) 2025-10-10T01:02:27.6647716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6648149Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6648582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6648990Z return func(*args, **kwargs) 2025-10-10T01:02:27.6649373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6649791Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6650197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6650624Z return func(*args, **kwargs) 2025-10-10T01:02:27.6651044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6651475Z value_states = self.v(current_states) 2025-10-10T01:02:27.6651622Z 2025-10-10T01:02:27.6651715Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6651954Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6652213Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6652613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6652979Z res = mod(**inputs) 2025-10-10T01:02:27.6653358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6653773Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6654183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6654636Z layer_outputs = layer_module( 2025-10-10T01:02:27.6655054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6655465Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6655890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6656317Z return func(*args, **kwargs) 2025-10-10T01:02:27.6656713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6657130Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6657557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6657972Z return func(*args, **kwargs) 2025-10-10T01:02:27.6658374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6658803Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6659224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6659645Z return func(*args, **kwargs) 2025-10-10T01:02:27.6660044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6660461Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6660608Z 2025-10-10T01:02:27.6660726Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6661133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6661499Z res = mod(**inputs) 2025-10-10T01:02:27.6661891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6662311Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6662717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6663133Z layer_outputs = layer_module( 2025-10-10T01:02:27.6663521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6663958Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6664455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6664880Z return func(*args, **kwargs) 2025-10-10T01:02:27.6665279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6665712Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6666153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6666728Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6667191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6667612Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6667765Z 2025-10-10T01:02:27.6667891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6668292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6668652Z res = mod(**inputs) 2025-10-10T01:02:27.6669045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6669465Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6669876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6671153Z layer_outputs = layer_module( 2025-10-10T01:02:27.6671540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6671933Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6672350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6672761Z return func(*args, **kwargs) 2025-10-10T01:02:27.6673150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6673580Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6674003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6674453Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6674895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6675299Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6675455Z 2025-10-10T01:02:27.6675570Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6675962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6676320Z res = mod(**inputs) 2025-10-10T01:02:27.6676690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6677093Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6677490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6677905Z layer_outputs = layer_module( 2025-10-10T01:02:27.6678284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6678690Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6679118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6679529Z return func(*args, **kwargs) 2025-10-10T01:02:27.6679948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6680380Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6680803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6681253Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6681718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6682136Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6682285Z 2025-10-10T01:02:27.6682400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6682806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6683161Z res = mod(**inputs) 2025-10-10T01:02:27.6683536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6683939Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6684333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6684737Z layer_outputs = layer_module( 2025-10-10T01:02:27.6685115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6685511Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6685943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6686368Z return func(*args, **kwargs) 2025-10-10T01:02:27.6686754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6687164Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6687578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6687974Z return func(*args, **kwargs) 2025-10-10T01:02:27.6688362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6688775Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6689193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6689598Z return func(*args, **kwargs) 2025-10-10T01:02:27.6689986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6690394Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6690548Z 2025-10-10T01:02:27.6690667Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6691069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6691424Z res = mod(**inputs) 2025-10-10T01:02:27.6691816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6692218Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6692612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6693012Z layer_outputs = layer_module( 2025-10-10T01:02:27.6693396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6693814Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6694229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6694640Z return func(*args, **kwargs) 2025-10-10T01:02:27.6695083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6695510Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6695938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6696368Z return func(*args, **kwargs) 2025-10-10T01:02:27.6696769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6697184Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6697602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6698027Z return func(*args, **kwargs) 2025-10-10T01:02:27.6698417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6698832Z key_states = self.k(current_states) 2025-10-10T01:02:27.6698981Z 2025-10-10T01:02:27.6699098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6699501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6699863Z res = mod(**inputs) 2025-10-10T01:02:27.6700248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6700656Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6701064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6701518Z layer_outputs = layer_module( 2025-10-10T01:02:27.6701913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6702324Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6702742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6703156Z return func(*args, **kwargs) 2025-10-10T01:02:27.6703552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6703973Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6704399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6704802Z return func(*args, **kwargs) 2025-10-10T01:02:27.6705202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6705631Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6706059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6706555Z return func(*args, **kwargs) 2025-10-10T01:02:27.6706963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6707388Z value_states = self.v(current_states) 2025-10-10T01:02:27.6707544Z 2025-10-10T01:02:27.6707650Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6707896Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6708158Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6708565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6708931Z res = mod(**inputs) 2025-10-10T01:02:27.6709328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6709738Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6710154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6710597Z layer_outputs = layer_module( 2025-10-10T01:02:27.6710981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6711385Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6711800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6712225Z return func(*args, **kwargs) 2025-10-10T01:02:27.6712622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6713045Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6713485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6713899Z return func(*args, **kwargs) 2025-10-10T01:02:27.6714298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6714731Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6715156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6715572Z return func(*args, **kwargs) 2025-10-10T01:02:27.6715969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6716393Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6716540Z 2025-10-10T01:02:27.6716667Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6717112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6717467Z res = mod(**inputs) 2025-10-10T01:02:27.6717851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6718266Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6718678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6719091Z layer_outputs = layer_module( 2025-10-10T01:02:27.6719483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6719885Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6720335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6720766Z return func(*args, **kwargs) 2025-10-10T01:02:27.6721172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6721604Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6722031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6722496Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6722952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6723379Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6723539Z 2025-10-10T01:02:27.6723653Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6724044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6724396Z res = mod(**inputs) 2025-10-10T01:02:27.6724794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6725214Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6725627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6726056Z layer_outputs = layer_module( 2025-10-10T01:02:27.6726448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6726865Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6727293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6727701Z return func(*args, **kwargs) 2025-10-10T01:02:27.6728089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6728551Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6728993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6729440Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6729883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6730295Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6730441Z 2025-10-10T01:02:27.6730562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6730945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6731298Z res = mod(**inputs) 2025-10-10T01:02:27.6731813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6732226Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6732719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6733114Z layer_outputs = layer_module( 2025-10-10T01:02:27.6733496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6733894Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6734309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6734720Z return func(*args, **kwargs) 2025-10-10T01:02:27.6735101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6735521Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6735938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6736389Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6736822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6737232Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6737389Z 2025-10-10T01:02:27.6737505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6737896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6738247Z res = mod(**inputs) 2025-10-10T01:02:27.6738618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6739022Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6739417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6739822Z layer_outputs = layer_module( 2025-10-10T01:02:27.6740199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6740582Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6740984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6741399Z return func(*args, **kwargs) 2025-10-10T01:02:27.6741765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6742161Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6742574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6742978Z return func(*args, **kwargs) 2025-10-10T01:02:27.6743369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6743811Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6744221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6744624Z return func(*args, **kwargs) 2025-10-10T01:02:27.6745012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6745413Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6745560Z 2025-10-10T01:02:27.6745683Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6746062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6746478Z res = mod(**inputs) 2025-10-10T01:02:27.6746858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6747268Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6747702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6748088Z layer_outputs = layer_module( 2025-10-10T01:02:27.6748452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6748829Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6749223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6749603Z return func(*args, **kwargs) 2025-10-10T01:02:27.6749970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6750361Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6750755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6751116Z return func(*args, **kwargs) 2025-10-10T01:02:27.6751470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6751843Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6752229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6752602Z return func(*args, **kwargs) 2025-10-10T01:02:27.6752950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6753330Z key_states = self.k(current_states) 2025-10-10T01:02:27.6753472Z 2025-10-10T01:02:27.6753577Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6753940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6754265Z res = mod(**inputs) 2025-10-10T01:02:27.6754618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6754983Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6755340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6755724Z layer_outputs = layer_module( 2025-10-10T01:02:27.6756066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6756437Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6756826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6757207Z return func(*args, **kwargs) 2025-10-10T01:02:27.6757578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6757972Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6758365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6758740Z return func(*args, **kwargs) 2025-10-10T01:02:27.6759106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6759484Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6759870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6760247Z return func(*args, **kwargs) 2025-10-10T01:02:27.6760609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6760988Z value_states = self.v(current_states) 2025-10-10T01:02:27.6761125Z 2025-10-10T01:02:27.6761211Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6761465Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6761707Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6762064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6762389Z res = mod(**inputs) 2025-10-10T01:02:27.6762729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6763094Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6763448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6763816Z layer_outputs = layer_module( 2025-10-10T01:02:27.6764156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6764519Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6764904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6765277Z return func(*args, **kwargs) 2025-10-10T01:02:27.6765625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6766005Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6766427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6766801Z return func(*args, **kwargs) 2025-10-10T01:02:27.6767157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6767537Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6767929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6768316Z return func(*args, **kwargs) 2025-10-10T01:02:27.6768683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6769068Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6769238Z 2025-10-10T01:02:27.6769344Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6769707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6770040Z res = mod(**inputs) 2025-10-10T01:02:27.6770398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6770774Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6771149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6771530Z layer_outputs = layer_module( 2025-10-10T01:02:27.6771919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6772285Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6772656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6773044Z return func(*args, **kwargs) 2025-10-10T01:02:27.6773431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6773839Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6774250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6774648Z return func(*args, **kwargs) 2025-10-10T01:02:27.6775033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-10-10T01:02:27.6775544Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T01:02:27.6775734Z 2025-10-10T01:02:27.6775852Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6776214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6776568Z res = mod(**inputs) 2025-10-10T01:02:27.6776942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6777323Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6777700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6778075Z layer_outputs = layer_module( 2025-10-10T01:02:27.6778430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6778814Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6779228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6779632Z return func(*args, **kwargs) 2025-10-10T01:02:27.6780012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6780434Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6780850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6781301Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6781736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6782142Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6782293Z 2025-10-10T01:02:27.6782409Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6782805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6783157Z res = mod(**inputs) 2025-10-10T01:02:27.6783534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6783968Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6784374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6784786Z layer_outputs = layer_module( 2025-10-10T01:02:27.6785168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6785574Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6785998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6786492Z return func(*args, **kwargs) 2025-10-10T01:02:27.6786941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6787377Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6787814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6788272Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6788718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6789129Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6789281Z 2025-10-10T01:02:27.6789400Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6789792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6790143Z res = mod(**inputs) 2025-10-10T01:02:27.6790544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6790972Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6791370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6791775Z layer_outputs = layer_module( 2025-10-10T01:02:27.6792153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6792553Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6792956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6793362Z return func(*args, **kwargs) 2025-10-10T01:02:27.6793753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6794178Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6794597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6795042Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6795460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6795844Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6795985Z 2025-10-10T01:02:27.6796102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6796460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6796792Z res = mod(**inputs) 2025-10-10T01:02:27.6797150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6797530Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6797927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6798317Z layer_outputs = layer_module( 2025-10-10T01:02:27.6798693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6799110Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6799525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6814660Z return func(*args, **kwargs) 2025-10-10T01:02:27.6815164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6815610Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6816048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6816496Z return func(*args, **kwargs) 2025-10-10T01:02:27.6816984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6817414Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6817841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6818242Z return func(*args, **kwargs) 2025-10-10T01:02:27.6818640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6819050Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6819203Z 2025-10-10T01:02:27.6819339Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6819750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6820105Z res = mod(**inputs) 2025-10-10T01:02:27.6820573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6820988Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6821396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6821793Z layer_outputs = layer_module( 2025-10-10T01:02:27.6822181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6822589Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6823009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6823422Z return func(*args, **kwargs) 2025-10-10T01:02:27.6823808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6824221Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6824647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6825054Z return func(*args, **kwargs) 2025-10-10T01:02:27.6825451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6825894Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6826420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6826864Z return func(*args, **kwargs) 2025-10-10T01:02:27.6827317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6827721Z key_states = self.k(current_states) 2025-10-10T01:02:27.6827878Z 2025-10-10T01:02:27.6828003Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6828426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6828839Z res = mod(**inputs) 2025-10-10T01:02:27.6829235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6829706Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6830136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6830562Z layer_outputs = layer_module( 2025-10-10T01:02:27.6830960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6831375Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6832014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6832445Z return func(*args, **kwargs) 2025-10-10T01:02:27.6832928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6833359Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6833787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6834202Z return func(*args, **kwargs) 2025-10-10T01:02:27.6834597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6835019Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6835442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6835856Z return func(*args, **kwargs) 2025-10-10T01:02:27.6836254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6836735Z value_states = self.v(current_states) 2025-10-10T01:02:27.6836895Z 2025-10-10T01:02:27.6836999Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6837236Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6837515Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6837909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6838272Z res = mod(**inputs) 2025-10-10T01:02:27.6838620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6839005Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6839381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6839765Z layer_outputs = layer_module( 2025-10-10T01:02:27.6840124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6840497Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6840884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6841266Z return func(*args, **kwargs) 2025-10-10T01:02:27.6841627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6842012Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6842393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6842783Z return func(*args, **kwargs) 2025-10-10T01:02:27.6843163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6843572Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6843983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6844387Z return func(*args, **kwargs) 2025-10-10T01:02:27.6844770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6845206Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6845353Z 2025-10-10T01:02:27.6845471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6845833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6846164Z res = mod(**inputs) 2025-10-10T01:02:27.6846517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6846896Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6847268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6847663Z layer_outputs = layer_module( 2025-10-10T01:02:27.6848023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6848403Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6848799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6849176Z return func(*args, **kwargs) 2025-10-10T01:02:27.6849547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6849956Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6850364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6850794Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6851258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6851660Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6851816Z 2025-10-10T01:02:27.6851933Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6852329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6852676Z res = mod(**inputs) 2025-10-10T01:02:27.6853054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6853460Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6853858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6854260Z layer_outputs = layer_module( 2025-10-10T01:02:27.6854630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6855030Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6855442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6855852Z return func(*args, **kwargs) 2025-10-10T01:02:27.6856246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6856665Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6857088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6857541Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6857984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6858387Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6858549Z 2025-10-10T01:02:27.6858665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6859057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6859432Z res = mod(**inputs) 2025-10-10T01:02:27.6859808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6860199Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6860592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6860994Z layer_outputs = layer_module( 2025-10-10T01:02:27.6861367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6861754Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6862183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6862591Z return func(*args, **kwargs) 2025-10-10T01:02:27.6862981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6863403Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6863809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6864255Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6864695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6865148Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6865296Z 2025-10-10T01:02:27.6865582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6865997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6866476Z res = mod(**inputs) 2025-10-10T01:02:27.6866862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6867274Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6867665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6868072Z layer_outputs = layer_module( 2025-10-10T01:02:27.6868454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6868854Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6869270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6869668Z return func(*args, **kwargs) 2025-10-10T01:02:27.6870066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6870468Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6870866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6871276Z return func(*args, **kwargs) 2025-10-10T01:02:27.6871657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6872071Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6872490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6872896Z return func(*args, **kwargs) 2025-10-10T01:02:27.6873275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6873688Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6873849Z 2025-10-10T01:02:27.6873969Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6874365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6874745Z res = mod(**inputs) 2025-10-10T01:02:27.6875116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6875527Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6875927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6876332Z layer_outputs = layer_module( 2025-10-10T01:02:27.6876703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6877112Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6877525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6877910Z return func(*args, **kwargs) 2025-10-10T01:02:27.6878277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6878679Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6879094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6879498Z return func(*args, **kwargs) 2025-10-10T01:02:27.6879887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6880298Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6880704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6881127Z return func(*args, **kwargs) 2025-10-10T01:02:27.6881538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6881941Z key_states = self.k(current_states) 2025-10-10T01:02:27.6882088Z 2025-10-10T01:02:27.6882207Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6882602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6882952Z res = mod(**inputs) 2025-10-10T01:02:27.6883328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6883732Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6884100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6884484Z layer_outputs = layer_module( 2025-10-10T01:02:27.6884847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6885226Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6885610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6885991Z return func(*args, **kwargs) 2025-10-10T01:02:27.6886356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6886736Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6887116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6887506Z return func(*args, **kwargs) 2025-10-10T01:02:27.6887864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6888250Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6888634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6889013Z return func(*args, **kwargs) 2025-10-10T01:02:27.6889373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6889772Z value_states = self.v(current_states) 2025-10-10T01:02:27.6889911Z 2025-10-10T01:02:27.6889996Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6890220Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6890478Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6890864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6891213Z res = mod(**inputs) 2025-10-10T01:02:27.6891578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6891988Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6892401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6892799Z layer_outputs = layer_module( 2025-10-10T01:02:27.6893169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6893564Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6893972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6894369Z return func(*args, **kwargs) 2025-10-10T01:02:27.6894751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6895147Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6895570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6895997Z return func(*args, **kwargs) 2025-10-10T01:02:27.6896361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6896741Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6897132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6897510Z return func(*args, **kwargs) 2025-10-10T01:02:27.6897886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6898297Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6898442Z 2025-10-10T01:02:27.6898556Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6898946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6899309Z res = mod(**inputs) 2025-10-10T01:02:27.6899700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6900100Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6900489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6900886Z layer_outputs = layer_module( 2025-10-10T01:02:27.6901262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6901658Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6902061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6902480Z return func(*args, **kwargs) 2025-10-10T01:02:27.6902876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6903297Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6903719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6904148Z return func(*args, **kwargs) 2025-10-10T01:02:27.6904545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6904978Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6905404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6905817Z return func(*args, **kwargs) 2025-10-10T01:02:27.6906215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6906713Z key_states = self.k(current_states) 2025-10-10T01:02:27.6906868Z 2025-10-10T01:02:27.6906999Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6907432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6907788Z res = mod(**inputs) 2025-10-10T01:02:27.6908182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6908607Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6909022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6909445Z layer_outputs = layer_module( 2025-10-10T01:02:27.6909825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6910231Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6910685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6911163Z return func(*args, **kwargs) 2025-10-10T01:02:27.6911564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6911993Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6912433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6912843Z return func(*args, **kwargs) 2025-10-10T01:02:27.6913247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6913668Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6914102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6914517Z return func(*args, **kwargs) 2025-10-10T01:02:27.6914918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6915343Z value_states = self.v(current_states) 2025-10-10T01:02:27.6915493Z 2025-10-10T01:02:27.6915583Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6915826Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6916093Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6916501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6916846Z res = mod(**inputs) 2025-10-10T01:02:27.6917217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6917621Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6918016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6918408Z layer_outputs = layer_module( 2025-10-10T01:02:27.6918788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6919186Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6919596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6920021Z return func(*args, **kwargs) 2025-10-10T01:02:27.6920400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6920808Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6921217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6921617Z return func(*args, **kwargs) 2025-10-10T01:02:27.6922001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6922403Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6922828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6923230Z return func(*args, **kwargs) 2025-10-10T01:02:27.6923616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6924018Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6924168Z 2025-10-10T01:02:27.6924283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6924670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6925022Z res = mod(**inputs) 2025-10-10T01:02:27.6925393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6925802Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6926239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6926640Z layer_outputs = layer_module( 2025-10-10T01:02:27.6927015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6927414Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6927816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6928227Z return func(*args, **kwargs) 2025-10-10T01:02:27.6928612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.6929019Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.6929427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6929830Z return func(*args, **kwargs) 2025-10-10T01:02:27.6930213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.6930626Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.6931039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6931430Z return func(*args, **kwargs) 2025-10-10T01:02:27.6932029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6932441Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6932588Z 2025-10-10T01:02:27.6932713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6933097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6933451Z res = mod(**inputs) 2025-10-10T01:02:27.6933832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6934237Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6934635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6935083Z layer_outputs = layer_module( 2025-10-10T01:02:27.6935458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6935859Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6936247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6936625Z return func(*args, **kwargs) 2025-10-10T01:02:27.6936982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6937385Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6937829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6938234Z return func(*args, **kwargs) 2025-10-10T01:02:27.6938617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6939006Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6939395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6939837Z return func(*args, **kwargs) 2025-10-10T01:02:27.6940219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6940623Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6940771Z 2025-10-10T01:02:27.6940886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6941329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6941684Z res = mod(**inputs) 2025-10-10T01:02:27.6942048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6942454Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6942845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6943249Z layer_outputs = layer_module( 2025-10-10T01:02:27.6943625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6944014Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6944424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6944829Z return func(*args, **kwargs) 2025-10-10T01:02:27.6945232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6945651Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6946070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6946565Z return func(*args, **kwargs) 2025-10-10T01:02:27.6946961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-10-10T01:02:27.6947428Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T01:02:27.6947645Z 2025-10-10T01:02:27.6947755Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6948128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6948462Z res = mod(**inputs) 2025-10-10T01:02:27.6948823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6949208Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6949579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6949996Z layer_outputs = layer_module( 2025-10-10T01:02:27.6950353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6950724Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6951113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6951486Z return func(*args, **kwargs) 2025-10-10T01:02:27.6951851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6952251Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6952662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6953080Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6953499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6953880Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6954020Z 2025-10-10T01:02:27.6954138Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6954506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6954830Z res = mod(**inputs) 2025-10-10T01:02:27.6955193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6955563Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6955988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6956358Z layer_outputs = layer_module( 2025-10-10T01:02:27.6956700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6957067Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6957446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6957818Z return func(*args, **kwargs) 2025-10-10T01:02:27.6958178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6958580Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6958983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6959411Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6959840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6960239Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6960400Z 2025-10-10T01:02:27.6960516Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6960904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6961256Z res = mod(**inputs) 2025-10-10T01:02:27.6961631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-10-10T01:02:27.6962011Z encoder_outputs = self.encoder( 2025-10-10T01:02:27.6962383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6962764Z layer_outputs = layer_module( 2025-10-10T01:02:27.6963143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6963531Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6963945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6964369Z return func(*args, **kwargs) 2025-10-10T01:02:27.6964749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6965160Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6965567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6966021Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6966430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6966840Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6966981Z 2025-10-10T01:02:27.6967096Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6967453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6967784Z res = mod(**inputs) 2025-10-10T01:02:27.6968149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6968548Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6968935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6969334Z layer_outputs = layer_module( 2025-10-10T01:02:27.6969709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6970155Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6970568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6970965Z return func(*args, **kwargs) 2025-10-10T01:02:27.6971356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.6971769Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.6972184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6972582Z return func(*args, **kwargs) 2025-10-10T01:02:27.6972973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.6973391Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.6973813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6974221Z return func(*args, **kwargs) 2025-10-10T01:02:27.6974604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.6975055Z key_states = self.k(current_states) 2025-10-10T01:02:27.6975206Z 2025-10-10T01:02:27.6975320Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6975709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6976061Z res = mod(**inputs) 2025-10-10T01:02:27.6976427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6976830Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6977224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6977629Z layer_outputs = layer_module( 2025-10-10T01:02:27.6977998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6978394Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6978805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6979230Z return func(*args, **kwargs) 2025-10-10T01:02:27.6979619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.6980027Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.6980446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6980850Z return func(*args, **kwargs) 2025-10-10T01:02:27.6981242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.6981667Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.6982080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6982480Z return func(*args, **kwargs) 2025-10-10T01:02:27.6982861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.6982948Z value_states = self.v(current_states) 2025-10-10T01:02:27.6982952Z 2025-10-10T01:02:27.6983048Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6983134Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.6983249Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6983476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6983546Z res = mod(**inputs) 2025-10-10T01:02:27.6983828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6983927Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6984184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6984272Z layer_outputs = layer_module( 2025-10-10T01:02:27.6984521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6984620Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6984890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6984974Z return func(*args, **kwargs) 2025-10-10T01:02:27.6985235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.6985327Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.6985604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6985682Z return func(*args, **kwargs) 2025-10-10T01:02:27.6985946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.6986041Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.6986306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6986465Z return func(*args, **kwargs) 2025-10-10T01:02:27.6986730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.6986825Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.6986829Z 2025-10-10T01:02:27.6986950Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6987188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6987262Z res = mod(**inputs) 2025-10-10T01:02:27.6987525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6987643Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6987907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6987995Z layer_outputs = layer_module( 2025-10-10T01:02:27.6988243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6988334Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6988609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6988688Z return func(*args, **kwargs) 2025-10-10T01:02:27.6988978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6989087Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6989346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6989485Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6989744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.6989843Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.6989847Z 2025-10-10T01:02:27.6989961Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6990188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6990261Z res = mod(**inputs) 2025-10-10T01:02:27.6990546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6990654Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6990971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6991061Z layer_outputs = layer_module( 2025-10-10T01:02:27.6991305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6991393Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6991664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6991741Z return func(*args, **kwargs) 2025-10-10T01:02:27.6992010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6992115Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6992377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6992513Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6992773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.6992872Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.6992876Z 2025-10-10T01:02:27.6992991Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6993217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6993290Z res = mod(**inputs) 2025-10-10T01:02:27.6993563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6993653Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6993917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6994003Z layer_outputs = layer_module( 2025-10-10T01:02:27.6994246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6994353Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6994633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6994712Z return func(*args, **kwargs) 2025-10-10T01:02:27.6994978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.6995077Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.6995340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.6995470Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.6995755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.6995854Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.6995860Z 2025-10-10T01:02:27.6995977Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6996211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6996284Z res = mod(**inputs) 2025-10-10T01:02:27.6996558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.6996647Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.6996916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.6997003Z layer_outputs = layer_module( 2025-10-10T01:02:27.6997286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.6997375Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.6997656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6997735Z return func(*args, **kwargs) 2025-10-10T01:02:27.6998003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.6998094Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.6998376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6998454Z return func(*args, **kwargs) 2025-10-10T01:02:27.6998721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.6998828Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.6999115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.6999199Z return func(*args, **kwargs) 2025-10-10T01:02:27.6999458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.6999556Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.6999559Z 2025-10-10T01:02:27.6999677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.6999889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.6999966Z res = mod(**inputs) 2025-10-10T01:02:27.7000220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7000306Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7000562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7000639Z layer_outputs = layer_module( 2025-10-10T01:02:27.7000881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7001001Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7001264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7001339Z return func(*args, **kwargs) 2025-10-10T01:02:27.7001587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7001683Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7001939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7002024Z return func(*args, **kwargs) 2025-10-10T01:02:27.7002293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7002386Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7002650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7002724Z return func(*args, **kwargs) 2025-10-10T01:02:27.7002980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7003065Z key_states = self.k(current_states) 2025-10-10T01:02:27.7003069Z 2025-10-10T01:02:27.7003186Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7003397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7003468Z res = mod(**inputs) 2025-10-10T01:02:27.7003774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7003855Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7004116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7004199Z layer_outputs = layer_module( 2025-10-10T01:02:27.7004434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7004528Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7004783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7004865Z return func(*args, **kwargs) 2025-10-10T01:02:27.7005113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7005205Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7005472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7005549Z return func(*args, **kwargs) 2025-10-10T01:02:27.7005805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7005899Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7006164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7006238Z return func(*args, **kwargs) 2025-10-10T01:02:27.7006489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7006580Z value_states = self.v(current_states) 2025-10-10T01:02:27.7006584Z 2025-10-10T01:02:27.7006672Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7006769Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7006883Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7007097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7007194Z res = mod(**inputs) 2025-10-10T01:02:27.7007447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7007534Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7007785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7007863Z layer_outputs = layer_module( 2025-10-10T01:02:27.7008107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7008194Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7008460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7008553Z return func(*args, **kwargs) 2025-10-10T01:02:27.7008806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7008905Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7009160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7009243Z return func(*args, **kwargs) 2025-10-10T01:02:27.7009492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7009588Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7009844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7009917Z return func(*args, **kwargs) 2025-10-10T01:02:27.7010214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7010301Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7010305Z 2025-10-10T01:02:27.7010424Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7010637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7010709Z res = mod(**inputs) 2025-10-10T01:02:27.7010972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7011056Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7011323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7011403Z layer_outputs = layer_module( 2025-10-10T01:02:27.7011663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7011753Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7012010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7012094Z return func(*args, **kwargs) 2025-10-10T01:02:27.7012343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7012440Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7012697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7012771Z return func(*args, **kwargs) 2025-10-10T01:02:27.7013029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7013122Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7013392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7013467Z return func(*args, **kwargs) 2025-10-10T01:02:27.7013717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7013831Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7013835Z 2025-10-10T01:02:27.7013945Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7014163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7014234Z res = mod(**inputs) 2025-10-10T01:02:27.7014491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7014570Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7014823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7014924Z layer_outputs = layer_module( 2025-10-10T01:02:27.7015160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7015255Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7015521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7015597Z return func(*args, **kwargs) 2025-10-10T01:02:27.7015863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7015953Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7016224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7016299Z return func(*args, **kwargs) 2025-10-10T01:02:27.7016600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7016703Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7016969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7017054Z return func(*args, **kwargs) 2025-10-10T01:02:27.7017318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7017408Z key_states = self.k(current_states) 2025-10-10T01:02:27.7017412Z 2025-10-10T01:02:27.7017523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7017737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7017815Z res = mod(**inputs) 2025-10-10T01:02:27.7018082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7018175Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7018438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7018519Z layer_outputs = layer_module( 2025-10-10T01:02:27.7018772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7018860Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7019132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7019209Z return func(*args, **kwargs) 2025-10-10T01:02:27.7019467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7019563Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7019833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7019919Z return func(*args, **kwargs) 2025-10-10T01:02:27.7020178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7020298Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7020560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7020636Z return func(*args, **kwargs) 2025-10-10T01:02:27.7020903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7020991Z value_states = self.v(current_states) 2025-10-10T01:02:27.7020995Z 2025-10-10T01:02:27.7021092Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7021183Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7021300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7021543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7021619Z res = mod(**inputs) 2025-10-10T01:02:27.7021888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7021971Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7022232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7022318Z layer_outputs = layer_module( 2025-10-10T01:02:27.7022566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7022662Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7022948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7023058Z return func(*args, **kwargs) 2025-10-10T01:02:27.7023327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7023418Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7023698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7023775Z return func(*args, **kwargs) 2025-10-10T01:02:27.7024048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7024142Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7024411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7024495Z return func(*args, **kwargs) 2025-10-10T01:02:27.7024759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7024856Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7024860Z 2025-10-10T01:02:27.7024973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7025200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7025282Z res = mod(**inputs) 2025-10-10T01:02:27.7025549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7025639Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7025903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7025993Z layer_outputs = layer_module( 2025-10-10T01:02:27.7026240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7026410Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7026695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7026771Z return func(*args, **kwargs) 2025-10-10T01:02:27.7027060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7027164Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7027422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7027562Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7027818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.7027915Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.7027921Z 2025-10-10T01:02:27.7028039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7028285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7028363Z res = mod(**inputs) 2025-10-10T01:02:27.7028629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7028720Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7028982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7029069Z layer_outputs = layer_module( 2025-10-10T01:02:27.7029314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7029403Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7029698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7029799Z return func(*args, **kwargs) 2025-10-10T01:02:27.7030070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7030172Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7030437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7030575Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7030836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.7030933Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.7030936Z 2025-10-10T01:02:27.7031051Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7031276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7031353Z res = mod(**inputs) 2025-10-10T01:02:27.7031789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7031885Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7032151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7032238Z layer_outputs = layer_module( 2025-10-10T01:02:27.7032484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7032571Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7032859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7032938Z return func(*args, **kwargs) 2025-10-10T01:02:27.7033213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7033318Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7033579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7033761Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7034071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.7034172Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.7034177Z 2025-10-10T01:02:27.7034291Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7034516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7034589Z res = mod(**inputs) 2025-10-10T01:02:27.7034867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7034959Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7035279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7035369Z layer_outputs = layer_module( 2025-10-10T01:02:27.7035617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7035707Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7035984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7036065Z return func(*args, **kwargs) 2025-10-10T01:02:27.7036338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7036428Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7036753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7036876Z return func(*args, **kwargs) 2025-10-10T01:02:27.7037141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7037249Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7037521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7037605Z return func(*args, **kwargs) 2025-10-10T01:02:27.7037880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7037967Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7037971Z 2025-10-10T01:02:27.7038094Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7038316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7038401Z res = mod(**inputs) 2025-10-10T01:02:27.7038689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7038771Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7039064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7039143Z layer_outputs = layer_module( 2025-10-10T01:02:27.7039400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7039488Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7039780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7039855Z return func(*args, **kwargs) 2025-10-10T01:02:27.7040134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7040236Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7040525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7040626Z return func(*args, **kwargs) 2025-10-10T01:02:27.7040894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7040990Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7041269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7041346Z return func(*args, **kwargs) 2025-10-10T01:02:27.7041628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7041710Z key_states = self.k(current_states) 2025-10-10T01:02:27.7041716Z 2025-10-10T01:02:27.7041852Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7042074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7042144Z res = mod(**inputs) 2025-10-10T01:02:27.7042410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7042490Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7042763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7042842Z layer_outputs = layer_module( 2025-10-10T01:02:27.7043080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7043174Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7043450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7043552Z return func(*args, **kwargs) 2025-10-10T01:02:27.7043803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7043892Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7044161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7044238Z return func(*args, **kwargs) 2025-10-10T01:02:27.7044505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7044598Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7044872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7044954Z return func(*args, **kwargs) 2025-10-10T01:02:27.7045211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7045301Z value_states = self.v(current_states) 2025-10-10T01:02:27.7045305Z 2025-10-10T01:02:27.7045392Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7045485Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7045597Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7045810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7045888Z res = mod(**inputs) 2025-10-10T01:02:27.7046143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7046228Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7046482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7046561Z layer_outputs = layer_module( 2025-10-10T01:02:27.7046811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7046900Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7047163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7047257Z return func(*args, **kwargs) 2025-10-10T01:02:27.7047510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7047604Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7047862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7047944Z return func(*args, **kwargs) 2025-10-10T01:02:27.7048196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7048295Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7048569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7048643Z return func(*args, **kwargs) 2025-10-10T01:02:27.7048904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7048988Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7048992Z 2025-10-10T01:02:27.7049110Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7049323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7049394Z res = mod(**inputs) 2025-10-10T01:02:27.7049657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7049736Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7050044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7050123Z layer_outputs = layer_module( 2025-10-10T01:02:27.7050359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7050456Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7050713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7050794Z return func(*args, **kwargs) 2025-10-10T01:02:27.7051047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7051141Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7051399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7051478Z return func(*args, **kwargs) 2025-10-10T01:02:27.7051739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-10-10T01:02:27.7051884Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-10-10T01:02:27.7051890Z 2025-10-10T01:02:27.7052010Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7052227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7052301Z res = mod(**inputs) 2025-10-10T01:02:27.7052567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7052648Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7052916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7052999Z layer_outputs = layer_module( 2025-10-10T01:02:27.7053243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7053341Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7053616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7053719Z return func(*args, **kwargs) 2025-10-10T01:02:27.7053977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7054072Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7054336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7054410Z return func(*args, **kwargs) 2025-10-10T01:02:27.7054671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7054788Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7055055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7055128Z return func(*args, **kwargs) 2025-10-10T01:02:27.7055377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7055472Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7055476Z 2025-10-10T01:02:27.7055589Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7055807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7055878Z res = mod(**inputs) 2025-10-10T01:02:27.7056130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7056217Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7056505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7056593Z layer_outputs = layer_module( 2025-10-10T01:02:27.7056831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7056930Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7057194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7057271Z return func(*args, **kwargs) 2025-10-10T01:02:27.7057545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7057634Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7057901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7057978Z return func(*args, **kwargs) 2025-10-10T01:02:27.7058232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7058333Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7058593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7058674Z return func(*args, **kwargs) 2025-10-10T01:02:27.7058924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7059016Z key_states = self.k(current_states) 2025-10-10T01:02:27.7059020Z 2025-10-10T01:02:27.7059131Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7059341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7059419Z res = mod(**inputs) 2025-10-10T01:02:27.7059677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7059764Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7060017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7060112Z layer_outputs = layer_module( 2025-10-10T01:02:27.7060356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7060440Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7060707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7060782Z return func(*args, **kwargs) 2025-10-10T01:02:27.7061038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7061137Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7061420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7061506Z return func(*args, **kwargs) 2025-10-10T01:02:27.7061763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7061863Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7062126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7062201Z return func(*args, **kwargs) 2025-10-10T01:02:27.7062468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7062554Z value_states = self.v(current_states) 2025-10-10T01:02:27.7062558Z 2025-10-10T01:02:27.7062653Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7062777Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7062897Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7063122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7063196Z res = mod(**inputs) 2025-10-10T01:02:27.7063465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7063545Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7063805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7063892Z layer_outputs = layer_module( 2025-10-10T01:02:27.7064139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7064235Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7064510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7064585Z return func(*args, **kwargs) 2025-10-10T01:02:27.7064853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7064945Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7065221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7065299Z return func(*args, **kwargs) 2025-10-10T01:02:27.7065566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7065661Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7065930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7066017Z return func(*args, **kwargs) 2025-10-10T01:02:27.7066281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7066451Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7066480Z 2025-10-10T01:02:27.7066603Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7066820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7066904Z res = mod(**inputs) 2025-10-10T01:02:27.7067168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7067260Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7067521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7067612Z layer_outputs = layer_module( 2025-10-10T01:02:27.7067891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7067980Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7068245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7068324Z return func(*args, **kwargs) 2025-10-10T01:02:27.7068586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7068686Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7068938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7069075Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7069327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.7069458Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.7069465Z 2025-10-10T01:02:27.7069578Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7069789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7069868Z res = mod(**inputs) 2025-10-10T01:02:27.7070123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7070209Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7070463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7070545Z layer_outputs = layer_module( 2025-10-10T01:02:27.7070784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7070869Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7071139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7071213Z return func(*args, **kwargs) 2025-10-10T01:02:27.7071469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7071569Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7071819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7071952Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7072203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.7072298Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.7072302Z 2025-10-10T01:02:27.7072414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7072635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7072707Z res = mod(**inputs) 2025-10-10T01:02:27.7072960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7073092Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7073340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7073424Z layer_outputs = layer_module( 2025-10-10T01:02:27.7073661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7073747Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7074012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7074086Z return func(*args, **kwargs) 2025-10-10T01:02:27.7074376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7074477Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7074729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7074863Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7075116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.7075209Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.7075213Z 2025-10-10T01:02:27.7075325Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7075545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7075615Z res = mod(**inputs) 2025-10-10T01:02:27.7075906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7075994Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7076249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7076335Z layer_outputs = layer_module( 2025-10-10T01:02:27.7076569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7076654Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7076919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7076992Z return func(*args, **kwargs) 2025-10-10T01:02:27.7077248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7077339Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7077596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7077677Z return func(*args, **kwargs) 2025-10-10T01:02:27.7077925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7078024Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7078279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7078527Z return func(*args, **kwargs) 2025-10-10T01:02:27.7078779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7078864Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7078868Z 2025-10-10T01:02:27.7078988Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7079202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7079279Z res = mod(**inputs) 2025-10-10T01:02:27.7079550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7079649Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7079918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7079996Z layer_outputs = layer_module( 2025-10-10T01:02:27.7080240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7080328Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7080610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7080685Z return func(*args, **kwargs) 2025-10-10T01:02:27.7080960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7081064Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7081325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7081410Z return func(*args, **kwargs) 2025-10-10T01:02:27.7081673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7081765Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7082041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7082115Z return func(*args, **kwargs) 2025-10-10T01:02:27.7082404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7082525Z key_states = self.k(current_states) 2025-10-10T01:02:27.7082532Z 2025-10-10T01:02:27.7082652Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7082862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7082935Z res = mod(**inputs) 2025-10-10T01:02:27.7083200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7083281Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7083541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7083621Z layer_outputs = layer_module( 2025-10-10T01:02:27.7083865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7083963Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7084233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7084317Z return func(*args, **kwargs) 2025-10-10T01:02:27.7084576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7084669Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7084942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7085018Z return func(*args, **kwargs) 2025-10-10T01:02:27.7085283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7085384Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7085665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7085744Z return func(*args, **kwargs) 2025-10-10T01:02:27.7085997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7086089Z value_states = self.v(current_states) 2025-10-10T01:02:27.7086111Z 2025-10-10T01:02:27.7086199Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7086292Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7086404Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7086619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7086698Z res = mod(**inputs) 2025-10-10T01:02:27.7086953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7087042Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7087295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7087395Z layer_outputs = layer_module( 2025-10-10T01:02:27.7087643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7087734Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7088005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7088083Z return func(*args, **kwargs) 2025-10-10T01:02:27.7088341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7088439Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7088703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7088786Z return func(*args, **kwargs) 2025-10-10T01:02:27.7089086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7089190Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7089456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7089534Z return func(*args, **kwargs) 2025-10-10T01:02:27.7089802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7089888Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7089892Z 2025-10-10T01:02:27.7090016Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7090233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7090306Z res = mod(**inputs) 2025-10-10T01:02:27.7090578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7090662Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7090928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7091008Z layer_outputs = layer_module( 2025-10-10T01:02:27.7091251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7091347Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7091610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7091692Z return func(*args, **kwargs) 2025-10-10T01:02:27.7091946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7092044Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7092314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7092388Z return func(*args, **kwargs) 2025-10-10T01:02:27.7092650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7092762Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7093036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7093112Z return func(*args, **kwargs) 2025-10-10T01:02:27.7093377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7093470Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7093474Z 2025-10-10T01:02:27.7093589Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7093816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7093908Z res = mod(**inputs) 2025-10-10T01:02:27.7094177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7094258Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7094522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7094607Z layer_outputs = layer_module( 2025-10-10T01:02:27.7094854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7094951Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7095215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7095290Z return func(*args, **kwargs) 2025-10-10T01:02:27.7095577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7095686Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7095956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7096035Z return func(*args, **kwargs) 2025-10-10T01:02:27.7096292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7096394Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7096657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7096743Z return func(*args, **kwargs) 2025-10-10T01:02:27.7096999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7097092Z key_states = self.k(current_states) 2025-10-10T01:02:27.7097099Z 2025-10-10T01:02:27.7097216Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7097434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7097515Z res = mod(**inputs) 2025-10-10T01:02:27.7097777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7097866Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7098127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7098207Z layer_outputs = layer_module( 2025-10-10T01:02:27.7098458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7098547Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7098820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7098900Z return func(*args, **kwargs) 2025-10-10T01:02:27.7099157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7099284Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7099548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7099633Z return func(*args, **kwargs) 2025-10-10T01:02:27.7099891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7099992Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7100255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7100331Z return func(*args, **kwargs) 2025-10-10T01:02:27.7100619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7100710Z value_states = self.v(current_states) 2025-10-10T01:02:27.7100714Z 2025-10-10T01:02:27.7100809Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7100898Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7101012Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7101235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7101306Z res = mod(**inputs) 2025-10-10T01:02:27.7101571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7101652Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7101910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7102034Z layer_outputs = layer_module( 2025-10-10T01:02:27.7102281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7102379Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7102645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7102722Z return func(*args, **kwargs) 2025-10-10T01:02:27.7102986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7103077Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7103346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7103423Z return func(*args, **kwargs) 2025-10-10T01:02:27.7103687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7103783Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7104046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7104132Z return func(*args, **kwargs) 2025-10-10T01:02:27.7104388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7104482Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7104487Z 2025-10-10T01:02:27.7104601Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7104817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7104897Z res = mod(**inputs) 2025-10-10T01:02:27.7105157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7105248Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7105507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7105592Z layer_outputs = layer_module( 2025-10-10T01:02:27.7105858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7105946Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7106220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7106296Z return func(*args, **kwargs) 2025-10-10T01:02:27.7106670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7106767Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7107037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7107147Z return func(*args, **kwargs) 2025-10-10T01:02:27.7107411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 651, in forward 2025-10-10T01:02:27.7107569Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-10-10T01:02:27.7107574Z 2025-10-10T01:02:27.7107689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7107909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7107990Z res = mod(**inputs) 2025-10-10T01:02:27.7108256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7108344Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7108610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7109752Z layer_outputs = layer_module( 2025-10-10T01:02:27.7110006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7110094Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7110374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7110451Z return func(*args, **kwargs) 2025-10-10T01:02:27.7110718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7110822Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7111080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7111221Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7111484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.7111582Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.7111586Z 2025-10-10T01:02:27.7111703Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7111930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7112002Z res = mod(**inputs) 2025-10-10T01:02:27.7112262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7112350Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7112609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7112695Z layer_outputs = layer_module( 2025-10-10T01:02:27.7112938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7113029Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7113305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7113385Z return func(*args, **kwargs) 2025-10-10T01:02:27.7113675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7113776Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7114034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7114171Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7114426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.7114526Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.7114532Z 2025-10-10T01:02:27.7114647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7114893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7114968Z res = mod(**inputs) 2025-10-10T01:02:27.7115228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7115318Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7115594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7115682Z layer_outputs = layer_module( 2025-10-10T01:02:27.7115924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7116013Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7116313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7116410Z return func(*args, **kwargs) 2025-10-10T01:02:27.7116676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7116776Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7117045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7117181Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7117441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.7117540Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.7117544Z 2025-10-10T01:02:27.7117658Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7117881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7117958Z res = mod(**inputs) 2025-10-10T01:02:27.7118222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7118313Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7118575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7118660Z layer_outputs = layer_module( 2025-10-10T01:02:27.7118904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7118992Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7119277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7119354Z return func(*args, **kwargs) 2025-10-10T01:02:27.7119626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7119719Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7120009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7120104Z return func(*args, **kwargs) 2025-10-10T01:02:27.7120376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7120476Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7120755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7120839Z return func(*args, **kwargs) 2025-10-10T01:02:27.7121112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7121199Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7121205Z 2025-10-10T01:02:27.7121334Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7121572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7121653Z res = mod(**inputs) 2025-10-10T01:02:27.7121919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7122000Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7122283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7122362Z layer_outputs = layer_module( 2025-10-10T01:02:27.7122615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7122704Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7123006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7123114Z return func(*args, **kwargs) 2025-10-10T01:02:27.7123374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7123475Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7123752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7123836Z return func(*args, **kwargs) 2025-10-10T01:02:27.7124106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7124198Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7124482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7124558Z return func(*args, **kwargs) 2025-10-10T01:02:27.7124829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7124916Z key_states = self.k(current_states) 2025-10-10T01:02:27.7124920Z 2025-10-10T01:02:27.7125035Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7125261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7125336Z res = mod(**inputs) 2025-10-10T01:02:27.7125602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7125681Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7125950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7126030Z layer_outputs = layer_module( 2025-10-10T01:02:27.7126275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7126373Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7126639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7126722Z return func(*args, **kwargs) 2025-10-10T01:02:27.7126999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7127088Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7127360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7127436Z return func(*args, **kwargs) 2025-10-10T01:02:27.7127703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7127794Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7128076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7128163Z return func(*args, **kwargs) 2025-10-10T01:02:27.7128421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7128516Z value_states = self.v(current_states) 2025-10-10T01:02:27.7128520Z 2025-10-10T01:02:27.7128610Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7128706Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7128821Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7129039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7129120Z res = mod(**inputs) 2025-10-10T01:02:27.7129382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7129471Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7129771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7129854Z layer_outputs = layer_module( 2025-10-10T01:02:27.7130106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7130197Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7130471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7130549Z return func(*args, **kwargs) 2025-10-10T01:02:27.7130808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7130905Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7131169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7131258Z return func(*args, **kwargs) 2025-10-10T01:02:27.7131654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7131760Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7132029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7132106Z return func(*args, **kwargs) 2025-10-10T01:02:27.7132394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7132482Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7132487Z 2025-10-10T01:02:27.7132611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7132828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7132903Z res = mod(**inputs) 2025-10-10T01:02:27.7133179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7133262Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7133531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7133656Z layer_outputs = layer_module( 2025-10-10T01:02:27.7133900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7133997Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7134261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7134344Z return func(*args, **kwargs) 2025-10-10T01:02:27.7134600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7134703Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7134996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7135076Z return func(*args, **kwargs) 2025-10-10T01:02:27.7135349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7135446Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7135719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7135797Z return func(*args, **kwargs) 2025-10-10T01:02:27.7136057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7136151Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7136155Z 2025-10-10T01:02:27.7136298Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7136552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7136625Z res = mod(**inputs) 2025-10-10T01:02:27.7136901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7136990Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7137264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7137352Z layer_outputs = layer_module( 2025-10-10T01:02:27.7137643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7137740Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7138018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7138097Z return func(*args, **kwargs) 2025-10-10T01:02:27.7138371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7138462Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7138754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7138831Z return func(*args, **kwargs) 2025-10-10T01:02:27.7139094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7139194Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7139473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7139553Z return func(*args, **kwargs) 2025-10-10T01:02:27.7139828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7139915Z key_states = self.k(current_states) 2025-10-10T01:02:27.7139926Z 2025-10-10T01:02:27.7140039Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7140257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7140357Z res = mod(**inputs) 2025-10-10T01:02:27.7140633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7140719Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7140989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7141066Z layer_outputs = layer_module( 2025-10-10T01:02:27.7141310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7141399Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7141705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7141780Z return func(*args, **kwargs) 2025-10-10T01:02:27.7142028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7142127Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7142384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7142463Z return func(*args, **kwargs) 2025-10-10T01:02:27.7142712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7142801Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7143083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7143175Z return func(*args, **kwargs) 2025-10-10T01:02:27.7143437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7143523Z value_states = self.v(current_states) 2025-10-10T01:02:27.7143529Z 2025-10-10T01:02:27.7143622Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7143708Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7143819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7144043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7144114Z res = mod(**inputs) 2025-10-10T01:02:27.7144386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7144465Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7144730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7144818Z layer_outputs = layer_module( 2025-10-10T01:02:27.7145058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7145154Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7145413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7145487Z return func(*args, **kwargs) 2025-10-10T01:02:27.7145749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7145838Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7146108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7146185Z return func(*args, **kwargs) 2025-10-10T01:02:27.7146515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7146616Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7146874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7146980Z return func(*args, **kwargs) 2025-10-10T01:02:27.7147244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7147340Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7147345Z 2025-10-10T01:02:27.7147460Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7147684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7147766Z res = mod(**inputs) 2025-10-10T01:02:27.7148043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7148153Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7148410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7148492Z layer_outputs = layer_module( 2025-10-10T01:02:27.7148746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7148835Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7149101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7149175Z return func(*args, **kwargs) 2025-10-10T01:02:27.7149434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7149534Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7149835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7149974Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7150231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.7150331Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.7150335Z 2025-10-10T01:02:27.7150451Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7150666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7150745Z res = mod(**inputs) 2025-10-10T01:02:27.7151008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7151095Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7151358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7151445Z layer_outputs = layer_module( 2025-10-10T01:02:27.7151689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7151788Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7152054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7152129Z return func(*args, **kwargs) 2025-10-10T01:02:27.7152388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7152486Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7152737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7152873Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7153122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.7153217Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.7153238Z 2025-10-10T01:02:27.7153349Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7153569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7153639Z res = mod(**inputs) 2025-10-10T01:02:27.7153895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7153981Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7154237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7154321Z layer_outputs = layer_module( 2025-10-10T01:02:27.7154582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7154671Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7154939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7155019Z return func(*args, **kwargs) 2025-10-10T01:02:27.7155280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7155381Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7155635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7155769Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7156023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.7156150Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.7156156Z 2025-10-10T01:02:27.7156272Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7156494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7156568Z res = mod(**inputs) 2025-10-10T01:02:27.7156837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7156922Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7157173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7157257Z layer_outputs = layer_module( 2025-10-10T01:02:27.7157492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7157577Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7157843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7157918Z return func(*args, **kwargs) 2025-10-10T01:02:27.7158173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7158272Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7158519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 344, in forward 2025-10-10T01:02:27.7158667Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-10-10T01:02:27.7158671Z 2025-10-10T01:02:27.7158782Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7159001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7159071Z res = mod(**inputs) 2025-10-10T01:02:27.7159337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7159420Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7159678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7159788Z layer_outputs = layer_module( 2025-10-10T01:02:27.7160033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7160141Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7160397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7160472Z return func(*args, **kwargs) 2025-10-10T01:02:27.7160730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7160820Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7161112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7161191Z return func(*args, **kwargs) 2025-10-10T01:02:27.7161448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7161554Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7161822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7161907Z return func(*args, **kwargs) 2025-10-10T01:02:27.7162165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7162258Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7162262Z 2025-10-10T01:02:27.7162377Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7162631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7162712Z res = mod(**inputs) 2025-10-10T01:02:27.7162978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7163069Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7163335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7163414Z layer_outputs = layer_module( 2025-10-10T01:02:27.7163667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7163756Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7164027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7164104Z return func(*args, **kwargs) 2025-10-10T01:02:27.7164375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7164465Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7164732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7164819Z return func(*args, **kwargs) 2025-10-10T01:02:27.7165079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7165178Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7165444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7165519Z return func(*args, **kwargs) 2025-10-10T01:02:27.7165786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7165878Z key_states = self.k(current_states) 2025-10-10T01:02:27.7165883Z 2025-10-10T01:02:27.7166005Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7166225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7166318Z res = mod(**inputs) 2025-10-10T01:02:27.7166588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7166668Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7166933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7167013Z layer_outputs = layer_module( 2025-10-10T01:02:27.7167263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7167354Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7167638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7167728Z return func(*args, **kwargs) 2025-10-10T01:02:27.7167987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7168085Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7168351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7168426Z return func(*args, **kwargs) 2025-10-10T01:02:27.7168694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7168785Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7169057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7169174Z return func(*args, **kwargs) 2025-10-10T01:02:27.7169436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7169530Z value_states = self.v(current_states) 2025-10-10T01:02:27.7169535Z 2025-10-10T01:02:27.7169624Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7169722Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7169836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7170060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7170132Z res = mod(**inputs) 2025-10-10T01:02:27.7170392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7170482Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7170743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7170830Z layer_outputs = layer_module( 2025-10-10T01:02:27.7171074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7171165Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7171437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7171512Z return func(*args, **kwargs) 2025-10-10T01:02:27.7171778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-10-10T01:02:27.7171866Z self_attention_outputs = self.layer[0]( 2025-10-10T01:02:27.7172131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7172213Z return func(*args, **kwargs) 2025-10-10T01:02:27.7172476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-10-10T01:02:27.7172567Z attention_output = self.SelfAttention( 2025-10-10T01:02:27.7172808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7172904Z return func(*args, **kwargs) 2025-10-10T01:02:27.7173142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7173221Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7173224Z 2025-10-10T01:02:27.7173338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7173538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7173611Z res = mod(**inputs) 2025-10-10T01:02:27.7173851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7173954Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7174200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7174274Z layer_outputs = layer_module( 2025-10-10T01:02:27.7174505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7174586Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7174829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7174909Z return func(*args, **kwargs) 2025-10-10T01:02:27.7175142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7175234Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7175492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7175589Z return func(*args, **kwargs) 2025-10-10T01:02:27.7175834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7175923Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7176180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7176250Z return func(*args, **kwargs) 2025-10-10T01:02:27.7176500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-10-10T01:02:27.7176580Z query_states = self.q(hidden_states) 2025-10-10T01:02:27.7176583Z 2025-10-10T01:02:27.7176688Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7176901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7176970Z res = mod(**inputs) 2025-10-10T01:02:27.7177223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7177298Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7177546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7177628Z layer_outputs = layer_module( 2025-10-10T01:02:27.7177857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7177945Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7178193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7178270Z return func(*args, **kwargs) 2025-10-10T01:02:27.7178513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7178601Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7178858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7178952Z return func(*args, **kwargs) 2025-10-10T01:02:27.7179203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7179291Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7179539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7179616Z return func(*args, **kwargs) 2025-10-10T01:02:27.7179859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-10-10T01:02:27.7179946Z key_states = self.k(current_states) 2025-10-10T01:02:27.7179953Z 2025-10-10T01:02:27.7180083Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7180296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7180373Z res = mod(**inputs) 2025-10-10T01:02:27.7180627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7180714Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7180967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7181050Z layer_outputs = layer_module( 2025-10-10T01:02:27.7181286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7181371Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7181650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7181745Z return func(*args, **kwargs) 2025-10-10T01:02:27.7182008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7182098Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7182363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7182447Z return func(*args, **kwargs) 2025-10-10T01:02:27.7182706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7182803Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7183068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7183141Z return func(*args, **kwargs) 2025-10-10T01:02:27.7183408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-10-10T01:02:27.7183493Z value_states = self.v(current_states) 2025-10-10T01:02:27.7183497Z 2025-10-10T01:02:27.7183591Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7183681Z cudagraph partition due to non gpu ops 2025-10-10T01:02:27.7183801Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7184015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7184084Z res = mod(**inputs) 2025-10-10T01:02:27.7184355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7184433Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7184701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7184781Z layer_outputs = layer_module( 2025-10-10T01:02:27.7185025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7185119Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7185392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7185473Z return func(*args, **kwargs) 2025-10-10T01:02:27.7185722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-10-10T01:02:27.7185809Z cross_attention_outputs = self.layer[1]( 2025-10-10T01:02:27.7186075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7186149Z return func(*args, **kwargs) 2025-10-10T01:02:27.7186658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-10-10T01:02:27.7186787Z attention_output = self.EncDecAttention( 2025-10-10T01:02:27.7187060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7187139Z return func(*args, **kwargs) 2025-10-10T01:02:27.7187405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-10-10T01:02:27.7187501Z attn_output = self.o(attn_output) 2025-10-10T01:02:27.7187506Z 2025-10-10T01:02:27.7187620Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7187844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7187911Z res = mod(**inputs) 2025-10-10T01:02:27.7188150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7188267Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7188510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7188593Z layer_outputs = layer_module( 2025-10-10T01:02:27.7188822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7188910Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7189189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7189265Z return func(*args, **kwargs) 2025-10-10T01:02:27.7189534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7189635Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7189902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7190037Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7190295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-10-10T01:02:27.7190400Z hidden_states = self.wi(hidden_states) 2025-10-10T01:02:27.7190404Z 2025-10-10T01:02:27.7190518Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7190744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7190816Z res = mod(**inputs) 2025-10-10T01:02:27.7191077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7191165Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7191426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7191516Z layer_outputs = layer_module( 2025-10-10T01:02:27.7191762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7191852Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7192141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7192218Z return func(*args, **kwargs) 2025-10-10T01:02:27.7192481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7192581Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7192845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7192976Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7193234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-10-10T01:02:27.7193354Z hidden_states = self.act(hidden_states) 2025-10-10T01:02:27.7193358Z 2025-10-10T01:02:27.7193474Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7193699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7193770Z res = mod(**inputs) 2025-10-10T01:02:27.7194033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-10-10T01:02:27.7194123Z decoder_outputs = self.decoder( 2025-10-10T01:02:27.7194393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-10-10T01:02:27.7194474Z layer_outputs = layer_module( 2025-10-10T01:02:27.7194700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:02:27.7194820Z return super().__call__(*args, **kwargs) 2025-10-10T01:02:27.7195069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:02:27.7195140Z return func(*args, **kwargs) 2025-10-10T01:02:27.7195385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-10-10T01:02:27.7195476Z hidden_states = self.layer[-1](hidden_states) 2025-10-10T01:02:27.7195718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-10-10T01:02:27.7195834Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-10-10T01:02:27.7196068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-10-10T01:02:27.7196158Z hidden_states = self.wo(hidden_states) 2025-10-10T01:02:27.7196163Z 2025-10-10T01:02:27.7196271Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7196480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7196546Z res = mod(**inputs) 2025-10-10T01:02:27.7196793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1792, in forward 2025-10-10T01:02:27.7196882Z lm_logits = self.lm_head(sequence_output) 2025-10-10T01:02:27.7196886Z 2025-10-10T01:02:27.7196989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:02:27.7197195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:02:27.7197262Z res = mod(**inputs) 2025-10-10T01:02:27.7197509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1799, in forward 2025-10-10T01:02:27.7197657Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-10-10T01:02:27.7197663Z 2025-10-10T01:02:38.5525155Z Compilation time (from dynamo_timed): 18.721072299 2025-10-10T01:02:38.5681454Z pass 2025-10-10T01:02:38.5682618Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:38.5683832Z TIMING: _recursive_pre_grad_passes:0.01244 _recursive_joint_graph_passes:0.66233 _recursive_post_grad_passes:0.07518 async_compile.wait:0.71476 code_gen:10.12471 inductor_compile:11.48497 backend_compile:15.66256 gc:0.00109 entire_frame_compile:18.72107 total_wall_time:18.72107 2025-10-10T01:02:38.5684817Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:11515 | FakeTensor.__torch_dispatch__:4072 | ProxyTorchDispatchMode.__torch_dispatch__:3370 2025-10-10T01:02:38.5685351Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-10-10T01:02:41.3359967Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:02:41.3360919Z import pynvml # type: ignore[import] 2025-10-10T01:02:44.8095353Z 2025-10-10T01:02:45.7677223Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:02:45.7682393Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:02:45.7690040Z cpu eval T5Small 2025-10-10T01:02:46.7684432Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:47.1788489Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:02:47.6132063Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:01.0265997Z Compilation time (from dynamo_timed): 11.877647716 2025-10-10T01:03:01.0337507Z pass 2025-10-10T01:03:01.0338629Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:01.0339429Z TIMING: _recursive_pre_grad_passes:0.01298 async_compile.wait:0.00451 backend_compile:8.8351 gc:0.00026 entire_frame_compile:11.87765 total_wall_time:11.87765 2025-10-10T01:03:01.0340084Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-10-10T01:03:01.0341027Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-10-10T01:03:03.5066714Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:03:03.5067647Z import pynvml # type: ignore[import] 2025-10-10T01:03:07.0727936Z 2025-10-10T01:03:09.5894868Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:03:09.5895225Z loading model: 0it [00:02, ?it/s] 2025-10-10T01:03:09.5911318Z cpu eval TrOCRForCausalLM 2025-10-10T01:03:09.7661433Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-10-10T01:03:09.8103653Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:10.1286619Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:10.3833502Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:18.5823314Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5824008Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5824255Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5824533Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5824757Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5825103Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5825487Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5825817Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5826126Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5826607Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5827292Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5827527Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5827802Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5828239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5828645Z res = mod(**inputs) 2025-10-10T01:03:18.5829085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5829540Z outputs = self.model.decoder( 2025-10-10T01:03:18.5829973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5830486Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5830891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5831309Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5832023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5832503Z return func(*args, **kwargs) 2025-10-10T01:03:18.5832947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5833430Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5833893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5834374Z return func(*args, **kwargs) 2025-10-10T01:03:18.5834963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.5835458Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.5835650Z 2025-10-10T01:03:18.5835774Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5836189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5836558Z res = mod(**inputs) 2025-10-10T01:03:18.5836963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5837396Z outputs = self.model.decoder( 2025-10-10T01:03:18.5837826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5838267Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5838656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5839031Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5839418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5839803Z return func(*args, **kwargs) 2025-10-10T01:03:18.5840186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5840633Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5841072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5841477Z return func(*args, **kwargs) 2025-10-10T01:03:18.5841892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.5842323Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.5842486Z 2025-10-10T01:03:18.5842606Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5842979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5843309Z res = mod(**inputs) 2025-10-10T01:03:18.5843714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5844114Z outputs = self.model.decoder( 2025-10-10T01:03:18.5844522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5844951Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5845344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5845745Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5846212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5846627Z return func(*args, **kwargs) 2025-10-10T01:03:18.5847004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5847415Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5847810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5848180Z return func(*args, **kwargs) 2025-10-10T01:03:18.5848558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.5848955Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.5849116Z 2025-10-10T01:03:18.5849203Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5849423Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5849771Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5850105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5850716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5851117Z res = mod(**inputs) 2025-10-10T01:03:18.5851572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5852060Z outputs = self.model.decoder( 2025-10-10T01:03:18.5852456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5852849Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5853343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5853713Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5854097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5854487Z return func(*args, **kwargs) 2025-10-10T01:03:18.5854862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5855275Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5855762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5856129Z return func(*args, **kwargs) 2025-10-10T01:03:18.5856501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.5856902Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.5857042Z 2025-10-10T01:03:18.5857157Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5857514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5857833Z res = mod(**inputs) 2025-10-10T01:03:18.5858196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5858609Z outputs = self.model.decoder( 2025-10-10T01:03:18.5858989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5859371Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5859725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5860098Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5860492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5860875Z return func(*args, **kwargs) 2025-10-10T01:03:18.5861275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5861722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5861905Z 2025-10-10T01:03:18.5862013Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5862386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5862720Z res = mod(**inputs) 2025-10-10T01:03:18.5863090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5863501Z outputs = self.model.decoder( 2025-10-10T01:03:18.5863919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5864344Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5864744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5865168Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5865580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5865970Z return func(*args, **kwargs) 2025-10-10T01:03:18.5866498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5866968Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5867394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.5867770Z return self.act(input) 2025-10-10T01:03:18.5867891Z 2025-10-10T01:03:18.5868022Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5868384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5868725Z res = mod(**inputs) 2025-10-10T01:03:18.5869097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5869496Z outputs = self.model.decoder( 2025-10-10T01:03:18.5869887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5870277Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5870639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5871013Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5871404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5871786Z return func(*args, **kwargs) 2025-10-10T01:03:18.5872161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.5872567Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.5872715Z 2025-10-10T01:03:18.5872823Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5873214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5873534Z res = mod(**inputs) 2025-10-10T01:03:18.5873901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5874298Z outputs = self.model.decoder( 2025-10-10T01:03:18.5874683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5875077Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5875426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5875801Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5876208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5876588Z return func(*args, **kwargs) 2025-10-10T01:03:18.5876965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5877387Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5877797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5878178Z return func(*args, **kwargs) 2025-10-10T01:03:18.5878549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.5878978Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.5879197Z 2025-10-10T01:03:18.5879348Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5879713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5880037Z res = mod(**inputs) 2025-10-10T01:03:18.5880406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5880795Z outputs = self.model.decoder( 2025-10-10T01:03:18.5881186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5881577Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5881935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5882305Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5882686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5883057Z return func(*args, **kwargs) 2025-10-10T01:03:18.5883433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5883856Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5884272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5884680Z return func(*args, **kwargs) 2025-10-10T01:03:18.5885091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.5885509Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.5885647Z 2025-10-10T01:03:18.5885761Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5886114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5886435Z res = mod(**inputs) 2025-10-10T01:03:18.5886792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5887173Z outputs = self.model.decoder( 2025-10-10T01:03:18.5887561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5887945Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5888299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5888663Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5889042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5889410Z return func(*args, **kwargs) 2025-10-10T01:03:18.5889798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5890262Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5890699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5891107Z return func(*args, **kwargs) 2025-10-10T01:03:18.5891505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.5891940Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.5892105Z 2025-10-10T01:03:18.5892190Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5892412Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5892632Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5892870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5893250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5893597Z res = mod(**inputs) 2025-10-10T01:03:18.5893969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5894359Z outputs = self.model.decoder( 2025-10-10T01:03:18.5894750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5895156Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5895507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5895874Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5896261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5896641Z return func(*args, **kwargs) 2025-10-10T01:03:18.5897026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5897449Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5897849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5898236Z return func(*args, **kwargs) 2025-10-10T01:03:18.5898613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.5899019Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.5899161Z 2025-10-10T01:03:18.5899275Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5899635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5899964Z res = mod(**inputs) 2025-10-10T01:03:18.5900333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5900733Z outputs = self.model.decoder( 2025-10-10T01:03:18.5901138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5901584Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5901963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5902363Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5902823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5903194Z return func(*args, **kwargs) 2025-10-10T01:03:18.5903574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5904014Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5904202Z 2025-10-10T01:03:18.5904345Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5904733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5905083Z res = mod(**inputs) 2025-10-10T01:03:18.5905472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5905896Z outputs = self.model.decoder( 2025-10-10T01:03:18.5906392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5906822Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5907207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5907840Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5908269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5908673Z return func(*args, **kwargs) 2025-10-10T01:03:18.5909049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5909497Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5909896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.5910254Z return self.act(input) 2025-10-10T01:03:18.5910374Z 2025-10-10T01:03:18.5910493Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5910878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5911230Z res = mod(**inputs) 2025-10-10T01:03:18.5911619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5912031Z outputs = self.model.decoder( 2025-10-10T01:03:18.5912410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5912804Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5913170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5913545Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5913930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5914308Z return func(*args, **kwargs) 2025-10-10T01:03:18.5914689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.5915096Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.5915239Z 2025-10-10T01:03:18.5915357Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5915722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5916053Z res = mod(**inputs) 2025-10-10T01:03:18.5916419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5916829Z outputs = self.model.decoder( 2025-10-10T01:03:18.5917209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5917594Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5917954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5918322Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5918708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5919087Z return func(*args, **kwargs) 2025-10-10T01:03:18.5919477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5919944Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5920386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5920799Z return func(*args, **kwargs) 2025-10-10T01:03:18.5921208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.5921675Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.5921852Z 2025-10-10T01:03:18.5921960Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5922329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5922728Z res = mod(**inputs) 2025-10-10T01:03:18.5923094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5923516Z outputs = self.model.decoder( 2025-10-10T01:03:18.5923935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5924362Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5924738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5925133Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5925550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5925960Z return func(*args, **kwargs) 2025-10-10T01:03:18.5926374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5926826Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5927259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5927668Z return func(*args, **kwargs) 2025-10-10T01:03:18.5928080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.5928519Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.5928667Z 2025-10-10T01:03:18.5928781Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5929167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5929562Z res = mod(**inputs) 2025-10-10T01:03:18.5929944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5930361Z outputs = self.model.decoder( 2025-10-10T01:03:18.5930761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5931178Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5931730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5932143Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5932550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5932969Z return func(*args, **kwargs) 2025-10-10T01:03:18.5933374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5933832Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5934265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5934734Z return func(*args, **kwargs) 2025-10-10T01:03:18.5935135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.5935568Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.5935722Z 2025-10-10T01:03:18.5935820Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5936054Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5936285Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5936531Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5936901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5937233Z res = mod(**inputs) 2025-10-10T01:03:18.5937593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5938046Z outputs = self.model.decoder( 2025-10-10T01:03:18.5938438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5938829Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5939181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5939550Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5939935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5940314Z return func(*args, **kwargs) 2025-10-10T01:03:18.5940695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5941112Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5941518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5941899Z return func(*args, **kwargs) 2025-10-10T01:03:18.5942280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.5942681Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.5942823Z 2025-10-10T01:03:18.5942930Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5943297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5943627Z res = mod(**inputs) 2025-10-10T01:03:18.5943995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5944380Z outputs = self.model.decoder( 2025-10-10T01:03:18.5944780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5945200Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5945582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5945972Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5946461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5946877Z return func(*args, **kwargs) 2025-10-10T01:03:18.5947291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5947777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5947970Z 2025-10-10T01:03:18.5948087Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5948450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5948792Z res = mod(**inputs) 2025-10-10T01:03:18.5949204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5949607Z outputs = self.model.decoder( 2025-10-10T01:03:18.5949987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5950383Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5950742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5951116Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5951510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5951866Z return func(*args, **kwargs) 2025-10-10T01:03:18.5952247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5952685Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5953076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.5953421Z return self.act(input) 2025-10-10T01:03:18.5953532Z 2025-10-10T01:03:18.5953636Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5953989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5954310Z res = mod(**inputs) 2025-10-10T01:03:18.5954671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5955037Z outputs = self.model.decoder( 2025-10-10T01:03:18.5955401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5955774Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5956115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5956465Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5956829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5957187Z return func(*args, **kwargs) 2025-10-10T01:03:18.5957544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.5957925Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.5958058Z 2025-10-10T01:03:18.5958165Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5958503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5958813Z res = mod(**inputs) 2025-10-10T01:03:18.5959166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5959537Z outputs = self.model.decoder( 2025-10-10T01:03:18.5959894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5960285Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5960631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5960990Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5961364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5961726Z return func(*args, **kwargs) 2025-10-10T01:03:18.5962100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5962513Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5962926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5963284Z return func(*args, **kwargs) 2025-10-10T01:03:18.5963653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.5964066Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.5964227Z 2025-10-10T01:03:18.5964338Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5964682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5964986Z res = mod(**inputs) 2025-10-10T01:03:18.5965339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5965701Z outputs = self.model.decoder( 2025-10-10T01:03:18.5966093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5966452Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5966775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5967124Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5967492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5967852Z return func(*args, **kwargs) 2025-10-10T01:03:18.5968206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5968602Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5968986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5969351Z return func(*args, **kwargs) 2025-10-10T01:03:18.5969711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.5970088Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.5970225Z 2025-10-10T01:03:18.5970327Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5970679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5971000Z res = mod(**inputs) 2025-10-10T01:03:18.5971357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5971738Z outputs = self.model.decoder( 2025-10-10T01:03:18.5972112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5972494Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5972841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5973196Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5973585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5973979Z return func(*args, **kwargs) 2025-10-10T01:03:18.5974354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5974769Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5975173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5975540Z return func(*args, **kwargs) 2025-10-10T01:03:18.5975910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.5976321Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.5976464Z 2025-10-10T01:03:18.5976553Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5976763Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5976977Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.5977213Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5977569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5977884Z res = mod(**inputs) 2025-10-10T01:03:18.5978250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5978630Z outputs = self.model.decoder( 2025-10-10T01:03:18.5979001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5979401Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5979749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5980113Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5980530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5980903Z return func(*args, **kwargs) 2025-10-10T01:03:18.5981275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.5981697Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.5982109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5982487Z return func(*args, **kwargs) 2025-10-10T01:03:18.5982869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.5983269Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.5983417Z 2025-10-10T01:03:18.5983524Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5983893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5984221Z res = mod(**inputs) 2025-10-10T01:03:18.5984581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5984979Z outputs = self.model.decoder( 2025-10-10T01:03:18.5985369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5985764Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5986135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5986613Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5987034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5987439Z return func(*args, **kwargs) 2025-10-10T01:03:18.5987859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5988304Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5988479Z 2025-10-10T01:03:18.5988596Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5988946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5989262Z res = mod(**inputs) 2025-10-10T01:03:18.5989613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5989986Z outputs = self.model.decoder( 2025-10-10T01:03:18.5990371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5990749Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5991090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5991443Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5991805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5992172Z return func(*args, **kwargs) 2025-10-10T01:03:18.5992551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.5992995Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.5993403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.5993760Z return self.act(input) 2025-10-10T01:03:18.5993877Z 2025-10-10T01:03:18.5993980Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5994340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5994649Z res = mod(**inputs) 2025-10-10T01:03:18.5994988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5995363Z outputs = self.model.decoder( 2025-10-10T01:03:18.5995729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.5996105Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.5996444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.5996790Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.5997168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.5997537Z return func(*args, **kwargs) 2025-10-10T01:03:18.5997906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.5998288Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.5998433Z 2025-10-10T01:03:18.5998536Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.5998894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.5999204Z res = mod(**inputs) 2025-10-10T01:03:18.5999548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.5999913Z outputs = self.model.decoder( 2025-10-10T01:03:18.6000289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6000670Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6001019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6001431Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6001800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6002170Z return func(*args, **kwargs) 2025-10-10T01:03:18.6002542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6002952Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6003342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6003719Z return func(*args, **kwargs) 2025-10-10T01:03:18.6004108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6004548Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6004719Z 2025-10-10T01:03:18.6004836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6005196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6005527Z res = mod(**inputs) 2025-10-10T01:03:18.6005892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6006293Z outputs = self.model.decoder( 2025-10-10T01:03:18.6006668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6007044Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6007436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6007802Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6008193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6008583Z return func(*args, **kwargs) 2025-10-10T01:03:18.6008962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6009381Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6009786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6010161Z return func(*args, **kwargs) 2025-10-10T01:03:18.6010531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6010940Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6011085Z 2025-10-10T01:03:18.6011190Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6011550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6011875Z res = mod(**inputs) 2025-10-10T01:03:18.6012232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6012622Z outputs = self.model.decoder( 2025-10-10T01:03:18.6013006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6013445Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6013823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6014234Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6014669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6015097Z return func(*args, **kwargs) 2025-10-10T01:03:18.6015531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6016000Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6016408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6016784Z return func(*args, **kwargs) 2025-10-10T01:03:18.6017189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6017644Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6017804Z 2025-10-10T01:03:18.6017895Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6018138Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6018392Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6018663Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6019054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6019428Z res = mod(**inputs) 2025-10-10T01:03:18.6019837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6020278Z outputs = self.model.decoder( 2025-10-10T01:03:18.6020698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6021132Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6021521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6021964Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6022407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6022828Z return func(*args, **kwargs) 2025-10-10T01:03:18.6023300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6023774Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6024228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6024657Z return func(*args, **kwargs) 2025-10-10T01:03:18.6025081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6025550Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6025718Z 2025-10-10T01:03:18.6025833Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6026230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6026663Z res = mod(**inputs) 2025-10-10T01:03:18.6027070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6027522Z outputs = self.model.decoder( 2025-10-10T01:03:18.6027962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6028414Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6028793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6029191Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6029621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6030041Z return func(*args, **kwargs) 2025-10-10T01:03:18.6030443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6030927Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6031152Z 2025-10-10T01:03:18.6031268Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6031795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6032158Z res = mod(**inputs) 2025-10-10T01:03:18.6032544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6032987Z outputs = self.model.decoder( 2025-10-10T01:03:18.6033411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6033844Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6034292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6034675Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6035070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6035438Z return func(*args, **kwargs) 2025-10-10T01:03:18.6035806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6036233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6036611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6036954Z return self.act(input) 2025-10-10T01:03:18.6037068Z 2025-10-10T01:03:18.6037182Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6037605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6037923Z res = mod(**inputs) 2025-10-10T01:03:18.6038283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6038669Z outputs = self.model.decoder( 2025-10-10T01:03:18.6039047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6039428Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6039770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6040128Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6040506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6040876Z return func(*args, **kwargs) 2025-10-10T01:03:18.6041248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6041641Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6041784Z 2025-10-10T01:03:18.6041889Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6042244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6042563Z res = mod(**inputs) 2025-10-10T01:03:18.6042912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6043296Z outputs = self.model.decoder( 2025-10-10T01:03:18.6043682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6044077Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6044435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6044804Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6045195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6045605Z return func(*args, **kwargs) 2025-10-10T01:03:18.6045978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6046380Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6046778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6047152Z return func(*args, **kwargs) 2025-10-10T01:03:18.6047522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6047948Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6048113Z 2025-10-10T01:03:18.6048236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6048598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6048929Z res = mod(**inputs) 2025-10-10T01:03:18.6049293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6049684Z outputs = self.model.decoder( 2025-10-10T01:03:18.6050059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6050460Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6050837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6051207Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6051622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6052003Z return func(*args, **kwargs) 2025-10-10T01:03:18.6052366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6052765Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6053150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6053510Z return func(*args, **kwargs) 2025-10-10T01:03:18.6053889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6054275Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6054406Z 2025-10-10T01:03:18.6054517Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6054867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6055173Z res = mod(**inputs) 2025-10-10T01:03:18.6055522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6055897Z outputs = self.model.decoder( 2025-10-10T01:03:18.6056266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6056633Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6056973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6057325Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6057694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6058054Z return func(*args, **kwargs) 2025-10-10T01:03:18.6058413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6058811Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6059198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6059580Z return func(*args, **kwargs) 2025-10-10T01:03:18.6059944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6060344Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6060497Z 2025-10-10T01:03:18.6060581Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6060805Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6061024Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6061260Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6061658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6061988Z res = mod(**inputs) 2025-10-10T01:03:18.6062354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6062746Z outputs = self.model.decoder( 2025-10-10T01:03:18.6063141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6063562Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6063959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6064341Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6064729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6065122Z return func(*args, **kwargs) 2025-10-10T01:03:18.6065552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6065974Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6066453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6066840Z return func(*args, **kwargs) 2025-10-10T01:03:18.6067253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6067716Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6067875Z 2025-10-10T01:03:18.6068004Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6068403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6068750Z res = mod(**inputs) 2025-10-10T01:03:18.6069133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6069534Z outputs = self.model.decoder( 2025-10-10T01:03:18.6069928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6070320Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6070684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6071057Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6071450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6071839Z return func(*args, **kwargs) 2025-10-10T01:03:18.6072216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6072661Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6072848Z 2025-10-10T01:03:18.6072957Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6073328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6073676Z res = mod(**inputs) 2025-10-10T01:03:18.6074046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6074447Z outputs = self.model.decoder( 2025-10-10T01:03:18.6074837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6075234Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6075588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6075959Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6076367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6076752Z return func(*args, **kwargs) 2025-10-10T01:03:18.6077128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6077569Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6077968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6078307Z return self.act(input) 2025-10-10T01:03:18.6078414Z 2025-10-10T01:03:18.6078523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6078865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6079176Z res = mod(**inputs) 2025-10-10T01:03:18.6079544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6079947Z outputs = self.model.decoder( 2025-10-10T01:03:18.6080314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6080680Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6081025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6081372Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6081734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6082083Z return func(*args, **kwargs) 2025-10-10T01:03:18.6082446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6082828Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6082965Z 2025-10-10T01:03:18.6083077Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6083431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6083748Z res = mod(**inputs) 2025-10-10T01:03:18.6084115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6084492Z outputs = self.model.decoder( 2025-10-10T01:03:18.6084861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6085236Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6085567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6085918Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6086290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6086654Z return func(*args, **kwargs) 2025-10-10T01:03:18.6087009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6087429Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6087818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6088182Z return func(*args, **kwargs) 2025-10-10T01:03:18.6088544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6088950Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6089118Z 2025-10-10T01:03:18.6089227Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6089573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6089896Z res = mod(**inputs) 2025-10-10T01:03:18.6090244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6090627Z outputs = self.model.decoder( 2025-10-10T01:03:18.6091009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6091396Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6091751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6092112Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6092499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6092874Z return func(*args, **kwargs) 2025-10-10T01:03:18.6093288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6093686Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6094078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6094457Z return func(*args, **kwargs) 2025-10-10T01:03:18.6094831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6095225Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6095363Z 2025-10-10T01:03:18.6095473Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6095825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6096146Z res = mod(**inputs) 2025-10-10T01:03:18.6096510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6096904Z outputs = self.model.decoder( 2025-10-10T01:03:18.6097277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6097670Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6098024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6098390Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6098771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6099142Z return func(*args, **kwargs) 2025-10-10T01:03:18.6099515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6099926Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6100338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6100712Z return func(*args, **kwargs) 2025-10-10T01:03:18.6101097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6101522Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6101668Z 2025-10-10T01:03:18.6101759Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6101980Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6102191Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6102433Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6102798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6103128Z res = mod(**inputs) 2025-10-10T01:03:18.6103490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6103904Z outputs = self.model.decoder( 2025-10-10T01:03:18.6104294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6104685Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6105047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6105417Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6105807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6106190Z return func(*args, **kwargs) 2025-10-10T01:03:18.6106650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6107127Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6107571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6107978Z return func(*args, **kwargs) 2025-10-10T01:03:18.6108370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6108768Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6108907Z 2025-10-10T01:03:18.6109014Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6109373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6109694Z res = mod(**inputs) 2025-10-10T01:03:18.6110059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6110482Z outputs = self.model.decoder( 2025-10-10T01:03:18.6110898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6111323Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6111699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6112095Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6112469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6112837Z return func(*args, **kwargs) 2025-10-10T01:03:18.6113211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6113638Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6113808Z 2025-10-10T01:03:18.6113928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6114286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6114648Z res = mod(**inputs) 2025-10-10T01:03:18.6115048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6115497Z outputs = self.model.decoder( 2025-10-10T01:03:18.6115912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6116332Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6116710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6117102Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6117510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6117915Z return func(*args, **kwargs) 2025-10-10T01:03:18.6118339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6118807Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6119232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6119606Z return self.act(input) 2025-10-10T01:03:18.6119727Z 2025-10-10T01:03:18.6119842Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6120232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6120579Z res = mod(**inputs) 2025-10-10T01:03:18.6120970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6121399Z outputs = self.model.decoder( 2025-10-10T01:03:18.6121828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6122265Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6122624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6122997Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6123380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6123759Z return func(*args, **kwargs) 2025-10-10T01:03:18.6124138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6124537Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6124681Z 2025-10-10T01:03:18.6124795Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6125151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6125480Z res = mod(**inputs) 2025-10-10T01:03:18.6125844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6126234Z outputs = self.model.decoder( 2025-10-10T01:03:18.6126617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6126998Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6127354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6127713Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6128088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6128457Z return func(*args, **kwargs) 2025-10-10T01:03:18.6128835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6129255Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6129661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6130061Z return func(*args, **kwargs) 2025-10-10T01:03:18.6130433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6130872Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6131051Z 2025-10-10T01:03:18.6131160Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6131647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6131979Z res = mod(**inputs) 2025-10-10T01:03:18.6132336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6132777Z outputs = self.model.decoder( 2025-10-10T01:03:18.6133169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6133585Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6133961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6134355Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6134771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6135176Z return func(*args, **kwargs) 2025-10-10T01:03:18.6135571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6135993Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6136482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6136879Z return func(*args, **kwargs) 2025-10-10T01:03:18.6137262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6137677Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6137813Z 2025-10-10T01:03:18.6137919Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6138274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6138594Z res = mod(**inputs) 2025-10-10T01:03:18.6138952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6139329Z outputs = self.model.decoder( 2025-10-10T01:03:18.6139708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6140098Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6140462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6140838Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6141217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6141595Z return func(*args, **kwargs) 2025-10-10T01:03:18.6141979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6142399Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6142799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6143180Z return func(*args, **kwargs) 2025-10-10T01:03:18.6143568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6143998Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6144155Z 2025-10-10T01:03:18.6144252Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6144505Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6144733Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6144989Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6145375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6145716Z res = mod(**inputs) 2025-10-10T01:03:18.6146107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6146593Z outputs = self.model.decoder( 2025-10-10T01:03:18.6147022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6147498Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6147891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6148299Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6148693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6149085Z return func(*args, **kwargs) 2025-10-10T01:03:18.6149466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6149892Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6150307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6150691Z return func(*args, **kwargs) 2025-10-10T01:03:18.6151149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6151549Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6151700Z 2025-10-10T01:03:18.6151811Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6152178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6152510Z res = mod(**inputs) 2025-10-10T01:03:18.6152883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6153282Z outputs = self.model.decoder( 2025-10-10T01:03:18.6153703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6154106Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6154470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6154841Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6155226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6155604Z return func(*args, **kwargs) 2025-10-10T01:03:18.6155980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6156413Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6156584Z 2025-10-10T01:03:18.6156689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6157048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6157371Z res = mod(**inputs) 2025-10-10T01:03:18.6157735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6158124Z outputs = self.model.decoder( 2025-10-10T01:03:18.6158499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6158907Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6159256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6159616Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6159989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6160361Z return func(*args, **kwargs) 2025-10-10T01:03:18.6160737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6161208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6161611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6161951Z return self.act(input) 2025-10-10T01:03:18.6162072Z 2025-10-10T01:03:18.6162176Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6162538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6162860Z res = mod(**inputs) 2025-10-10T01:03:18.6163220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6163667Z outputs = self.model.decoder( 2025-10-10T01:03:18.6164047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6164437Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6164801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6165176Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6165554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6165923Z return func(*args, **kwargs) 2025-10-10T01:03:18.6166297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6166692Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6166831Z 2025-10-10T01:03:18.6166937Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6167293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6167613Z res = mod(**inputs) 2025-10-10T01:03:18.6167970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6168361Z outputs = self.model.decoder( 2025-10-10T01:03:18.6168735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6169139Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6169490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6169853Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6170234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6170623Z return func(*args, **kwargs) 2025-10-10T01:03:18.6170994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6171410Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6171815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6172202Z return func(*args, **kwargs) 2025-10-10T01:03:18.6172572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6173012Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6173179Z 2025-10-10T01:03:18.6173292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6173646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6173957Z res = mod(**inputs) 2025-10-10T01:03:18.6174317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6174699Z outputs = self.model.decoder( 2025-10-10T01:03:18.6175078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6175458Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6175823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6176192Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6176574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6176947Z return func(*args, **kwargs) 2025-10-10T01:03:18.6177310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6177716Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6178112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6178486Z return func(*args, **kwargs) 2025-10-10T01:03:18.6178867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6179271Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6179414Z 2025-10-10T01:03:18.6179520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6179904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6180250Z res = mod(**inputs) 2025-10-10T01:03:18.6180627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6181053Z outputs = self.model.decoder( 2025-10-10T01:03:18.6181458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6181883Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6182262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6182655Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6183065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6183469Z return func(*args, **kwargs) 2025-10-10T01:03:18.6183874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6184320Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6184750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6185154Z return func(*args, **kwargs) 2025-10-10T01:03:18.6185559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6186002Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6186158Z 2025-10-10T01:03:18.6186247Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6186564Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6186803Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6187068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6187490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6187918Z res = mod(**inputs) 2025-10-10T01:03:18.6188313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6188747Z outputs = self.model.decoder( 2025-10-10T01:03:18.6189155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6189573Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6189957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6190371Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6190793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6191205Z return func(*args, **kwargs) 2025-10-10T01:03:18.6191607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6192059Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6192496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6192904Z return func(*args, **kwargs) 2025-10-10T01:03:18.6193303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6193738Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6193932Z 2025-10-10T01:03:18.6194050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6194438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6194790Z res = mod(**inputs) 2025-10-10T01:03:18.6195174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6195596Z outputs = self.model.decoder( 2025-10-10T01:03:18.6195996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6196385Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6196737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6197115Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6197496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6197866Z return func(*args, **kwargs) 2025-10-10T01:03:18.6198238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6198658Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6198834Z 2025-10-10T01:03:18.6198939Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6199294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6199617Z res = mod(**inputs) 2025-10-10T01:03:18.6199974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6200350Z outputs = self.model.decoder( 2025-10-10T01:03:18.6200729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6201117Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6201472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6201840Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6202264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6202646Z return func(*args, **kwargs) 2025-10-10T01:03:18.6203033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6203476Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6203879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6204228Z return self.act(input) 2025-10-10T01:03:18.6204347Z 2025-10-10T01:03:18.6204453Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6204827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6205146Z res = mod(**inputs) 2025-10-10T01:03:18.6205493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6205879Z outputs = self.model.decoder( 2025-10-10T01:03:18.6206254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6206636Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6206987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6207361Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6207767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6208171Z return func(*args, **kwargs) 2025-10-10T01:03:18.6208565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6208954Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6209101Z 2025-10-10T01:03:18.6209205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6209566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6209887Z res = mod(**inputs) 2025-10-10T01:03:18.6210236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6210630Z outputs = self.model.decoder( 2025-10-10T01:03:18.6210898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6210987Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6211217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6211298Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6211538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6211617Z return func(*args, **kwargs) 2025-10-10T01:03:18.6211868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6211976Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6212217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6212289Z return func(*args, **kwargs) 2025-10-10T01:03:18.6212554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6212672Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6212677Z 2025-10-10T01:03:18.6212798Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6213013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6213078Z res = mod(**inputs) 2025-10-10T01:03:18.6213333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6213407Z outputs = self.model.decoder( 2025-10-10T01:03:18.6213660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6213734Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6213958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6214040Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6214294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6214372Z return func(*args, **kwargs) 2025-10-10T01:03:18.6214633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6214740Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6214990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6215062Z return func(*args, **kwargs) 2025-10-10T01:03:18.6215324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6215409Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6215412Z 2025-10-10T01:03:18.6215558Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6215761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6215833Z res = mod(**inputs) 2025-10-10T01:03:18.6216088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6216165Z outputs = self.model.decoder( 2025-10-10T01:03:18.6216425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6216500Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6216731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6216814Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6217059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6217140Z return func(*args, **kwargs) 2025-10-10T01:03:18.6217393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6217500Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6217745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6217816Z return func(*args, **kwargs) 2025-10-10T01:03:18.6218078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6218168Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6218172Z 2025-10-10T01:03:18.6218263Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6218343Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6218430Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6218539Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6218741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6218815Z res = mod(**inputs) 2025-10-10T01:03:18.6219068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6219181Z outputs = self.model.decoder( 2025-10-10T01:03:18.6219436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6219511Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6219746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6219829Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6220087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6220178Z return func(*args, **kwargs) 2025-10-10T01:03:18.6220436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6220562Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6220818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6220900Z return func(*args, **kwargs) 2025-10-10T01:03:18.6221168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6221264Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6221268Z 2025-10-10T01:03:18.6221379Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6221590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6221724Z res = mod(**inputs) 2025-10-10T01:03:18.6222005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6222094Z outputs = self.model.decoder( 2025-10-10T01:03:18.6222371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6222459Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6222707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6222792Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6223059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6223135Z return func(*args, **kwargs) 2025-10-10T01:03:18.6223406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6223546Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6223550Z 2025-10-10T01:03:18.6223662Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6223886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6223956Z res = mod(**inputs) 2025-10-10T01:03:18.6224234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6224315Z outputs = self.model.decoder( 2025-10-10T01:03:18.6224584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6224672Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6224912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6225011Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6225273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6225349Z return func(*args, **kwargs) 2025-10-10T01:03:18.6225649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6225778Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6226015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6226092Z return self.act(input) 2025-10-10T01:03:18.6226096Z 2025-10-10T01:03:18.6226214Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6226503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6226583Z res = mod(**inputs) 2025-10-10T01:03:18.6226885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6226966Z outputs = self.model.decoder( 2025-10-10T01:03:18.6227242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6227322Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6227560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6227657Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6227917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6228001Z return func(*args, **kwargs) 2025-10-10T01:03:18.6228272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6228419Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6228434Z 2025-10-10T01:03:18.6228547Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6228760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6228838Z res = mod(**inputs) 2025-10-10T01:03:18.6229108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6229195Z outputs = self.model.decoder( 2025-10-10T01:03:18.6229465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6229539Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6229770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6229853Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6230110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6230180Z return func(*args, **kwargs) 2025-10-10T01:03:18.6230435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6230548Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6230807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6230891Z return func(*args, **kwargs) 2025-10-10T01:03:18.6231161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6231282Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6231292Z 2025-10-10T01:03:18.6231405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6231759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6231846Z res = mod(**inputs) 2025-10-10T01:03:18.6232119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6233189Z outputs = self.model.decoder( 2025-10-10T01:03:18.6233452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6233529Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6233776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6233864Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6234131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6234211Z return func(*args, **kwargs) 2025-10-10T01:03:18.6234519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6234637Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6234894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6234979Z return func(*args, **kwargs) 2025-10-10T01:03:18.6235249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6235344Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6235348Z 2025-10-10T01:03:18.6235460Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6235671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6235779Z res = mod(**inputs) 2025-10-10T01:03:18.6236072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6236162Z outputs = self.model.decoder( 2025-10-10T01:03:18.6236431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6236511Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6236758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6236843Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6237112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6237187Z return func(*args, **kwargs) 2025-10-10T01:03:18.6237459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6237579Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6237846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6237928Z return func(*args, **kwargs) 2025-10-10T01:03:18.6238207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6238309Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6238313Z 2025-10-10T01:03:18.6238401Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6238487Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6238578Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6238690Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6238915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6238989Z res = mod(**inputs) 2025-10-10T01:03:18.6239272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6239360Z outputs = self.model.decoder( 2025-10-10T01:03:18.6239639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6239742Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6239987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6240075Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6240350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6240427Z return func(*args, **kwargs) 2025-10-10T01:03:18.6240716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6240841Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6241110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6241188Z return func(*args, **kwargs) 2025-10-10T01:03:18.6241458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6241555Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6241559Z 2025-10-10T01:03:18.6241670Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6241886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6241956Z res = mod(**inputs) 2025-10-10T01:03:18.6242226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6242363Z outputs = self.model.decoder( 2025-10-10T01:03:18.6242622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6242702Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6242927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6243011Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6243264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6243335Z return func(*args, **kwargs) 2025-10-10T01:03:18.6243595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6243714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6243718Z 2025-10-10T01:03:18.6243831Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6244035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6244101Z res = mod(**inputs) 2025-10-10T01:03:18.6244360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6244437Z outputs = self.model.decoder( 2025-10-10T01:03:18.6244694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6244766Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6244988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6245075Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6245317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6245397Z return func(*args, **kwargs) 2025-10-10T01:03:18.6245652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6245776Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6246018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6246090Z return self.act(input) 2025-10-10T01:03:18.6246094Z 2025-10-10T01:03:18.6246205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6246405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6246479Z res = mod(**inputs) 2025-10-10T01:03:18.6246736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6246810Z outputs = self.model.decoder( 2025-10-10T01:03:18.6247098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6247174Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6247405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6247487Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6247731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6247810Z return func(*args, **kwargs) 2025-10-10T01:03:18.6248062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6248153Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6248157Z 2025-10-10T01:03:18.6248262Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6248505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6248574Z res = mod(**inputs) 2025-10-10T01:03:18.6248829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6248915Z outputs = self.model.decoder( 2025-10-10T01:03:18.6249171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6249256Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6249480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6249560Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6249814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6249885Z return func(*args, **kwargs) 2025-10-10T01:03:18.6250153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6250251Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6250496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6250578Z return func(*args, **kwargs) 2025-10-10T01:03:18.6250834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-10-10T01:03:18.6250955Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:18.6250959Z 2025-10-10T01:03:18.6251063Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6251271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6251338Z res = mod(**inputs) 2025-10-10T01:03:18.6251599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6251681Z outputs = self.model.decoder( 2025-10-10T01:03:18.6251933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6252033Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6252258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6252340Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6252594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6252665Z return func(*args, **kwargs) 2025-10-10T01:03:18.6252925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6253028Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6253298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6253371Z return func(*args, **kwargs) 2025-10-10T01:03:18.6253624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-10-10T01:03:18.6253716Z key_states = self.k_proj(current_states) 2025-10-10T01:03:18.6253719Z 2025-10-10T01:03:18.6253824Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6254028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6254095Z res = mod(**inputs) 2025-10-10T01:03:18.6254349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6254431Z outputs = self.model.decoder( 2025-10-10T01:03:18.6254738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6254820Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6255049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6255132Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6255387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6255458Z return func(*args, **kwargs) 2025-10-10T01:03:18.6255722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6255823Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6256075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6256149Z return func(*args, **kwargs) 2025-10-10T01:03:18.6256407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-10-10T01:03:18.6256504Z value_states = self.v_proj(current_states) 2025-10-10T01:03:18.6256508Z 2025-10-10T01:03:18.6256590Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6256677Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6256758Z cudagraph partition due to non gpu ops 2025-10-10T01:03:18.6256865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6257076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6257142Z res = mod(**inputs) 2025-10-10T01:03:18.6257407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6257482Z outputs = self.model.decoder( 2025-10-10T01:03:18.6257742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6257825Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6258053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6258160Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6258405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6258481Z return func(*args, **kwargs) 2025-10-10T01:03:18.6258735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-10-10T01:03:18.6258834Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:18.6259088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6259161Z return func(*args, **kwargs) 2025-10-10T01:03:18.6259441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-10-10T01:03:18.6259525Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:18.6259530Z 2025-10-10T01:03:18.6259631Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6259830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6259895Z res = mod(**inputs) 2025-10-10T01:03:18.6260152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6260226Z outputs = self.model.decoder( 2025-10-10T01:03:18.6260479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6260590Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6260817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6260905Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6261148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6261228Z return func(*args, **kwargs) 2025-10-10T01:03:18.6261481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6261602Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6261606Z 2025-10-10T01:03:18.6261719Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6261923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6262000Z res = mod(**inputs) 2025-10-10T01:03:18.6262275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6262355Z outputs = self.model.decoder( 2025-10-10T01:03:18.6262632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6262712Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6262956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6263042Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6263304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6263380Z return func(*args, **kwargs) 2025-10-10T01:03:18.6263648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-10-10T01:03:18.6263785Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:18.6264015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:18.6264100Z return self.act(input) 2025-10-10T01:03:18.6264122Z 2025-10-10T01:03:18.6264235Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6264444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6264521Z res = mod(**inputs) 2025-10-10T01:03:18.6264790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-10-10T01:03:18.6264876Z outputs = self.model.decoder( 2025-10-10T01:03:18.6265144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-10-10T01:03:18.6265223Z layer_outputs = decoder_layer( 2025-10-10T01:03:18.6265487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:18.6265573Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:18.6265838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:18.6265915Z return func(*args, **kwargs) 2025-10-10T01:03:18.6266191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-10-10T01:03:18.6266280Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:18.6266284Z 2025-10-10T01:03:18.6266470Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6266694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6266766Z res = mod(**inputs) 2025-10-10T01:03:18.6267076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 843, in forward 2025-10-10T01:03:18.6267201Z logits = self.output_projection(outputs[0]) 2025-10-10T01:03:18.6267206Z 2025-10-10T01:03:18.6267324Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:18.6267551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:18.6267619Z res = mod(**inputs) 2025-10-10T01:03:18.6267876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 848, in forward 2025-10-10T01:03:18.6268022Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T01:03:18.6268026Z 2025-10-10T01:03:29.0710294Z Compilation time (from dynamo_timed): 17.338380393 2025-10-10T01:03:29.0749619Z pass 2025-10-10T01:03:29.0750101Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:29.0751033Z TIMING: _recursive_pre_grad_passes:0.00803 _recursive_joint_graph_passes:0.5072 _recursive_post_grad_passes:0.07473 async_compile.wait:0.82949 code_gen:9.51276 inductor_compile:10.84983 backend_compile:14.47357 gc:0.00142 entire_frame_compile:17.33838 total_wall_time:17.33838 2025-10-10T01:03:29.0752049Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:8431 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:2501 2025-10-10T01:03:29.0752620Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-10-10T01:03:31.8023555Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:03:31.8024434Z import pynvml # type: ignore[import] 2025-10-10T01:03:35.3664814Z 2025-10-10T01:03:41.8498700Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:03:41.8499036Z loading model: 0it [00:06, ?it/s] 2025-10-10T01:03:41.8521205Z cpu eval XGLMForCausalLM 2025-10-10T01:03:42.2531872Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-10-10T01:03:42.3385895Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:42.9643382Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:43.4697639Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:03:58.7109543Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7109999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7110377Z res = mod(**inputs) 2025-10-10T01:03:58.7110815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7111583Z outputs = self.model( 2025-10-10T01:03:58.7112012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7112471Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7112873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7113284Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7113717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7114143Z return func(*args, **kwargs) 2025-10-10T01:03:58.7114560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7115011Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7115591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7116032Z return func(*args, **kwargs) 2025-10-10T01:03:58.7116446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7116909Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7117109Z 2025-10-10T01:03:58.7117236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7128841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7129255Z res = mod(**inputs) 2025-10-10T01:03:58.7129707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7130151Z outputs = self.model( 2025-10-10T01:03:58.7130592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7131032Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7131430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7132041Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7132490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7132921Z return func(*args, **kwargs) 2025-10-10T01:03:58.7133330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7133767Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7134186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7134579Z return func(*args, **kwargs) 2025-10-10T01:03:58.7134976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7135386Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7135534Z 2025-10-10T01:03:58.7135659Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7136214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7136572Z res = mod(**inputs) 2025-10-10T01:03:58.7136968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7137363Z outputs = self.model( 2025-10-10T01:03:58.7137741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7138143Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7138511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7138958Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7139381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7139785Z return func(*args, **kwargs) 2025-10-10T01:03:58.7140193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7140640Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7141081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7141488Z return func(*args, **kwargs) 2025-10-10T01:03:58.7141896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7142448Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7142673Z 2025-10-10T01:03:58.7142810Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7143231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7143591Z res = mod(**inputs) 2025-10-10T01:03:58.7143997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7144436Z outputs = self.model( 2025-10-10T01:03:58.7144838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7145264Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7145652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7146071Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7146693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7147114Z return func(*args, **kwargs) 2025-10-10T01:03:58.7147521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7147983Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7148406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7148817Z return func(*args, **kwargs) 2025-10-10T01:03:58.7149226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7149716Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7149938Z 2025-10-10T01:03:58.7150059Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7150471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7150832Z res = mod(**inputs) 2025-10-10T01:03:58.7151231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7151669Z outputs = self.model( 2025-10-10T01:03:58.7152068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7152495Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7152887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7153285Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7153710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7154504Z return func(*args, **kwargs) 2025-10-10T01:03:58.7154966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7155493Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7155935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7156359Z return func(*args, **kwargs) 2025-10-10T01:03:58.7156741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7157163Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7157319Z 2025-10-10T01:03:58.7157446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7157832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7158185Z res = mod(**inputs) 2025-10-10T01:03:58.7158594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7159043Z outputs = self.model( 2025-10-10T01:03:58.7159412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7159817Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7160186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7160568Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7160966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7161352Z return func(*args, **kwargs) 2025-10-10T01:03:58.7161763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7162221Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7162650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7163038Z return func(*args, **kwargs) 2025-10-10T01:03:58.7163417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7163844Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7164008Z 2025-10-10T01:03:58.7164116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7164486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7164816Z res = mod(**inputs) 2025-10-10T01:03:58.7165186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7165585Z outputs = self.model( 2025-10-10T01:03:58.7165966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7166370Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7166732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7167125Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7167515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7167896Z return func(*args, **kwargs) 2025-10-10T01:03:58.7168280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7168688Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7169100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7169484Z return func(*args, **kwargs) 2025-10-10T01:03:58.7169886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7170337Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7170532Z 2025-10-10T01:03:58.7170642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7171029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7171365Z res = mod(**inputs) 2025-10-10T01:03:58.7171734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7172117Z outputs = self.model( 2025-10-10T01:03:58.7172487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7172879Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7173317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7173713Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7174116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7174503Z return func(*args, **kwargs) 2025-10-10T01:03:58.7174880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7175301Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7175702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7176088Z return func(*args, **kwargs) 2025-10-10T01:03:58.7176481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7176893Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7177037Z 2025-10-10T01:03:58.7177156Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7177515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7177850Z res = mod(**inputs) 2025-10-10T01:03:58.7178223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7178643Z outputs = self.model( 2025-10-10T01:03:58.7179045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7179465Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7179844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7180249Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7180684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7181092Z return func(*args, **kwargs) 2025-10-10T01:03:58.7181502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7182003Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7182199Z 2025-10-10T01:03:58.7182327Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7182729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7183086Z res = mod(**inputs) 2025-10-10T01:03:58.7183490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7183919Z outputs = self.model( 2025-10-10T01:03:58.7184324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7184770Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7185158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7185570Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7185996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7186506Z return func(*args, **kwargs) 2025-10-10T01:03:58.7186931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7187415Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7187864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7188239Z return self.act(input) 2025-10-10T01:03:58.7188404Z 2025-10-10T01:03:58.7188538Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7188925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7189273Z res = mod(**inputs) 2025-10-10T01:03:58.7189653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7190072Z outputs = self.model( 2025-10-10T01:03:58.7190470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7190882Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7191261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7191649Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7192062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7192470Z return func(*args, **kwargs) 2025-10-10T01:03:58.7192876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7193299Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7193449Z 2025-10-10T01:03:58.7193562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7193950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7194295Z res = mod(**inputs) 2025-10-10T01:03:58.7194675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7195071Z outputs = self.model( 2025-10-10T01:03:58.7195458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7195873Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7196250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7196643Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7197048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7197468Z return func(*args, **kwargs) 2025-10-10T01:03:58.7197867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7198321Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7198764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7199160Z return func(*args, **kwargs) 2025-10-10T01:03:58.7199562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7200043Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7200223Z 2025-10-10T01:03:58.7200346Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7200729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7201079Z res = mod(**inputs) 2025-10-10T01:03:58.7201463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7201875Z outputs = self.model( 2025-10-10T01:03:58.7202259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7202665Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7203047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7203485Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7203904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7204299Z return func(*args, **kwargs) 2025-10-10T01:03:58.7204699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7205141Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7205571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7205985Z return func(*args, **kwargs) 2025-10-10T01:03:58.7206375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7206810Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7206966Z 2025-10-10T01:03:58.7207086Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7207480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7207830Z res = mod(**inputs) 2025-10-10T01:03:58.7208207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7208618Z outputs = self.model( 2025-10-10T01:03:58.7209006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7209417Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7209789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7210183Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7210589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7210995Z return func(*args, **kwargs) 2025-10-10T01:03:58.7211395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7211824Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7212278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7212682Z return func(*args, **kwargs) 2025-10-10T01:03:58.7213087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7213550Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7213732Z 2025-10-10T01:03:58.7213853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7214247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7214606Z res = mod(**inputs) 2025-10-10T01:03:58.7215016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7215416Z outputs = self.model( 2025-10-10T01:03:58.7215802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7216216Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7216598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7216992Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7217394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7217795Z return func(*args, **kwargs) 2025-10-10T01:03:58.7218196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7218677Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7219106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7219500Z return func(*args, **kwargs) 2025-10-10T01:03:58.7219897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7220376Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7220581Z 2025-10-10T01:03:58.7220702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7221087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7221434Z res = mod(**inputs) 2025-10-10T01:03:58.7221820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7222234Z outputs = self.model( 2025-10-10T01:03:58.7222624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7223030Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7223414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7223809Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7224219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7224626Z return func(*args, **kwargs) 2025-10-10T01:03:58.7225013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7225451Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7225883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7226370Z return func(*args, **kwargs) 2025-10-10T01:03:58.7226778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7227244Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7227409Z 2025-10-10T01:03:58.7227527Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7227924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7228258Z res = mod(**inputs) 2025-10-10T01:03:58.7228614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7229004Z outputs = self.model( 2025-10-10T01:03:58.7229365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7229755Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7230139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7230512Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7230905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7231286Z return func(*args, **kwargs) 2025-10-10T01:03:58.7231821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7232235Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7232641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7233020Z return func(*args, **kwargs) 2025-10-10T01:03:58.7233438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7233882Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7234038Z 2025-10-10T01:03:58.7234146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7234514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7234847Z res = mod(**inputs) 2025-10-10T01:03:58.7235217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7235665Z outputs = self.model( 2025-10-10T01:03:58.7236041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7236436Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7236808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7237181Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7237563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7237946Z return func(*args, **kwargs) 2025-10-10T01:03:58.7238330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7238738Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7239138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7239508Z return func(*args, **kwargs) 2025-10-10T01:03:58.7239868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7240306Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7240487Z 2025-10-10T01:03:58.7240600Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7240963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7241299Z res = mod(**inputs) 2025-10-10T01:03:58.7241721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7242106Z outputs = self.model( 2025-10-10T01:03:58.7242471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7242834Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7243174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7243525Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7243892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7244282Z return func(*args, **kwargs) 2025-10-10T01:03:58.7244650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7245057Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7245456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7245826Z return func(*args, **kwargs) 2025-10-10T01:03:58.7246184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7246577Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7246723Z 2025-10-10T01:03:58.7246828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7247188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7247547Z res = mod(**inputs) 2025-10-10T01:03:58.7247906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7248294Z outputs = self.model( 2025-10-10T01:03:58.7248664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7249069Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7249415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7249780Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7250166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7250554Z return func(*args, **kwargs) 2025-10-10T01:03:58.7250945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7251372Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7251555Z 2025-10-10T01:03:58.7251660Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7252020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7252349Z res = mod(**inputs) 2025-10-10T01:03:58.7252719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7253085Z outputs = self.model( 2025-10-10T01:03:58.7253449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7253835Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7254188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7254552Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7254939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7255308Z return func(*args, **kwargs) 2025-10-10T01:03:58.7255695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7256128Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7256533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7256892Z return self.act(input) 2025-10-10T01:03:58.7257018Z 2025-10-10T01:03:58.7257132Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7257524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7257868Z res = mod(**inputs) 2025-10-10T01:03:58.7258257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7258645Z outputs = self.model( 2025-10-10T01:03:58.7259019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7259399Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7259741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7260105Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7260483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7260855Z return func(*args, **kwargs) 2025-10-10T01:03:58.7261224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7261639Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7261785Z 2025-10-10T01:03:58.7261891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7262247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7262566Z res = mod(**inputs) 2025-10-10T01:03:58.7262909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7263289Z outputs = self.model( 2025-10-10T01:03:58.7263651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7264038Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7264393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7264757Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7265145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7265521Z return func(*args, **kwargs) 2025-10-10T01:03:58.7265896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7266388Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7266808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7267210Z return func(*args, **kwargs) 2025-10-10T01:03:58.7267609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7268079Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7268248Z 2025-10-10T01:03:58.7268364Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7268741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7269069Z res = mod(**inputs) 2025-10-10T01:03:58.7269432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7269839Z outputs = self.model( 2025-10-10T01:03:58.7270203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7270598Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7270963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7271343Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7271732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7272110Z return func(*args, **kwargs) 2025-10-10T01:03:58.7272511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7272927Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7273331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7273700Z return func(*args, **kwargs) 2025-10-10T01:03:58.7274074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7274476Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7274616Z 2025-10-10T01:03:58.7274732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7275097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7275419Z res = mod(**inputs) 2025-10-10T01:03:58.7275795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7276198Z outputs = self.model( 2025-10-10T01:03:58.7276566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7276957Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7277312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7277687Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7278077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7278455Z return func(*args, **kwargs) 2025-10-10T01:03:58.7278823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7279240Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7279653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7280030Z return func(*args, **kwargs) 2025-10-10T01:03:58.7280402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7280824Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7280998Z 2025-10-10T01:03:58.7281107Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7281482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7281800Z res = mod(**inputs) 2025-10-10T01:03:58.7282147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7282524Z outputs = self.model( 2025-10-10T01:03:58.7282882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7283264Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7283614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7284002Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7284380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7284751Z return func(*args, **kwargs) 2025-10-10T01:03:58.7285118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7285520Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7285910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7286286Z return func(*args, **kwargs) 2025-10-10T01:03:58.7286670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7287109Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7287299Z 2025-10-10T01:03:58.7287413Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7287766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7288087Z res = mod(**inputs) 2025-10-10T01:03:58.7288441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7288823Z outputs = self.model( 2025-10-10T01:03:58.7289182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7289577Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7289962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7290326Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7290703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7291064Z return func(*args, **kwargs) 2025-10-10T01:03:58.7291441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7291854Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7292257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7292631Z return func(*args, **kwargs) 2025-10-10T01:03:58.7292994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7293390Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7293534Z 2025-10-10T01:03:58.7293646Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7294002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7294316Z res = mod(**inputs) 2025-10-10T01:03:58.7294668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7295041Z outputs = self.model( 2025-10-10T01:03:58.7295394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7295772Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7296113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7296472Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7296853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7297222Z return func(*args, **kwargs) 2025-10-10T01:03:58.7297583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7297995Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7298381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7298790Z return func(*args, **kwargs) 2025-10-10T01:03:58.7299156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7299550Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7299706Z 2025-10-10T01:03:58.7299809Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7300185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7300504Z res = mod(**inputs) 2025-10-10T01:03:58.7300855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7301226Z outputs = self.model( 2025-10-10T01:03:58.7301584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7301960Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7302308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7302662Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7303044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7303413Z return func(*args, **kwargs) 2025-10-10T01:03:58.7303815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7304226Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7304616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7304986Z return func(*args, **kwargs) 2025-10-10T01:03:58.7305353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7305790Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7305975Z 2025-10-10T01:03:58.7306098Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7306559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7306921Z res = mod(**inputs) 2025-10-10T01:03:58.7307332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7307745Z outputs = self.model( 2025-10-10T01:03:58.7308125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7308539Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7308914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7309278Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7309657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7310024Z return func(*args, **kwargs) 2025-10-10T01:03:58.7310394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7310798Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7311198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7311560Z return func(*args, **kwargs) 2025-10-10T01:03:58.7311949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7312339Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7312476Z 2025-10-10T01:03:58.7312589Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7312941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7313253Z res = mod(**inputs) 2025-10-10T01:03:58.7313605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7313977Z outputs = self.model( 2025-10-10T01:03:58.7314350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7314728Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7315076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7315441Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7315833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7316265Z return func(*args, **kwargs) 2025-10-10T01:03:58.7316631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7317061Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7317243Z 2025-10-10T01:03:58.7317350Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7317751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7318078Z res = mod(**inputs) 2025-10-10T01:03:58.7318425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7318809Z outputs = self.model( 2025-10-10T01:03:58.7319217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7319636Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7320016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7320402Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7320813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7321193Z return func(*args, **kwargs) 2025-10-10T01:03:58.7321577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7322005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7322407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7322760Z return self.act(input) 2025-10-10T01:03:58.7322875Z 2025-10-10T01:03:58.7322993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7323397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7323739Z res = mod(**inputs) 2025-10-10T01:03:58.7324102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7324492Z outputs = self.model( 2025-10-10T01:03:58.7324859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7325248Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7325605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7325996Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7326388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7326773Z return func(*args, **kwargs) 2025-10-10T01:03:58.7327141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7327539Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7327687Z 2025-10-10T01:03:58.7327794Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7328160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7328492Z res = mod(**inputs) 2025-10-10T01:03:58.7328866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7329262Z outputs = self.model( 2025-10-10T01:03:58.7329645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7330057Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7330427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7330825Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7331237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7331787Z return func(*args, **kwargs) 2025-10-10T01:03:58.7332194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7332646Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7332803Z 2025-10-10T01:03:58.7332915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7333298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7333648Z res = mod(**inputs) 2025-10-10T01:03:58.7334023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7334438Z outputs = self.model( 2025-10-10T01:03:58.7334825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7335250Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7335631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7336024Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7336443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7336843Z return func(*args, **kwargs) 2025-10-10T01:03:58.7337239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7337679Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7338105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7338505Z return func(*args, **kwargs) 2025-10-10T01:03:58.7338871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7339293Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7339457Z 2025-10-10T01:03:58.7339572Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7339927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7340248Z res = mod(**inputs) 2025-10-10T01:03:58.7340599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7341005Z outputs = self.model( 2025-10-10T01:03:58.7341351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7341740Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7342097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7342467Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7342851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7343227Z return func(*args, **kwargs) 2025-10-10T01:03:58.7343626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7344046Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7344456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7344831Z return func(*args, **kwargs) 2025-10-10T01:03:58.7345206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7345604Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7345745Z 2025-10-10T01:03:58.7345868Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7346308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7346661Z res = mod(**inputs) 2025-10-10T01:03:58.7347092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7347499Z outputs = self.model( 2025-10-10T01:03:58.7347876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7348264Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7348610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7348978Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7349356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7349725Z return func(*args, **kwargs) 2025-10-10T01:03:58.7350085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7350494Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7350892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7351259Z return func(*args, **kwargs) 2025-10-10T01:03:58.7351624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7352035Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7352212Z 2025-10-10T01:03:58.7352320Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7352688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7353023Z res = mod(**inputs) 2025-10-10T01:03:58.7353367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7353743Z outputs = self.model( 2025-10-10T01:03:58.7354104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7354484Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7354836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7355212Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7355587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7355956Z return func(*args, **kwargs) 2025-10-10T01:03:58.7356325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7356732Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7357123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7357493Z return func(*args, **kwargs) 2025-10-10T01:03:58.7357886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7358346Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7358536Z 2025-10-10T01:03:58.7358647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7359000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7359318Z res = mod(**inputs) 2025-10-10T01:03:58.7359672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7360051Z outputs = self.model( 2025-10-10T01:03:58.7360402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7360811Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7361164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7361528Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7361937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7362306Z return func(*args, **kwargs) 2025-10-10T01:03:58.7362676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7363084Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7363483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7363849Z return func(*args, **kwargs) 2025-10-10T01:03:58.7364218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7364613Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7364755Z 2025-10-10T01:03:58.7364869Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7365226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7365540Z res = mod(**inputs) 2025-10-10T01:03:58.7365893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7366266Z outputs = self.model( 2025-10-10T01:03:58.7366621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7366996Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7367336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7367699Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7368083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7368459Z return func(*args, **kwargs) 2025-10-10T01:03:58.7368847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7369270Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7369702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7370103Z return func(*args, **kwargs) 2025-10-10T01:03:58.7370481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7370891Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7371055Z 2025-10-10T01:03:58.7371175Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7371554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7371870Z res = mod(**inputs) 2025-10-10T01:03:58.7372215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7372579Z outputs = self.model( 2025-10-10T01:03:58.7372931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7373315Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7373673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7374040Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7374434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7374863Z return func(*args, **kwargs) 2025-10-10T01:03:58.7375236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7375645Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7376042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7376423Z return func(*args, **kwargs) 2025-10-10T01:03:58.7376797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7377237Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7377416Z 2025-10-10T01:03:58.7377527Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7377885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7378214Z res = mod(**inputs) 2025-10-10T01:03:58.7378587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7378981Z outputs = self.model( 2025-10-10T01:03:58.7379348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7379748Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7380113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7380493Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7380887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7381267Z return func(*args, **kwargs) 2025-10-10T01:03:58.7381652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7382076Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7382486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7382879Z return func(*args, **kwargs) 2025-10-10T01:03:58.7383257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7383680Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7383827Z 2025-10-10T01:03:58.7383948Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7384331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7384671Z res = mod(**inputs) 2025-10-10T01:03:58.7385053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7385460Z outputs = self.model( 2025-10-10T01:03:58.7385864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7386347Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7386731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7387144Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7387573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7387958Z return func(*args, **kwargs) 2025-10-10T01:03:58.7388332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7388776Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7388964Z 2025-10-10T01:03:58.7389102Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7389489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7389823Z res = mod(**inputs) 2025-10-10T01:03:58.7390179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7390565Z outputs = self.model( 2025-10-10T01:03:58.7390928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7391319Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7391680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7392045Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7392434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7392818Z return func(*args, **kwargs) 2025-10-10T01:03:58.7393201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7393639Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7394042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7394395Z return self.act(input) 2025-10-10T01:03:58.7394509Z 2025-10-10T01:03:58.7394623Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7394991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7395314Z res = mod(**inputs) 2025-10-10T01:03:58.7395675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7396064Z outputs = self.model( 2025-10-10T01:03:58.7396434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7396828Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7397211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7397623Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7398012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7398393Z return func(*args, **kwargs) 2025-10-10T01:03:58.7398760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7399158Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7399305Z 2025-10-10T01:03:58.7399413Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7399778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7400128Z res = mod(**inputs) 2025-10-10T01:03:58.7400491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7400867Z outputs = self.model( 2025-10-10T01:03:58.7401222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7401601Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7401943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7402303Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7402681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7403088Z return func(*args, **kwargs) 2025-10-10T01:03:58.7403511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7403940Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7404351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7404735Z return func(*args, **kwargs) 2025-10-10T01:03:58.7405114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7405552Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7405723Z 2025-10-10T01:03:58.7405831Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7406221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7406575Z res = mod(**inputs) 2025-10-10T01:03:58.7406941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7407330Z outputs = self.model( 2025-10-10T01:03:58.7407701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7408097Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7408470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7408865Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7409282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7409692Z return func(*args, **kwargs) 2025-10-10T01:03:58.7410088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7410542Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7410974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7411355Z return func(*args, **kwargs) 2025-10-10T01:03:58.7411730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7412148Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7412289Z 2025-10-10T01:03:58.7412402Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7412769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7413119Z res = mod(**inputs) 2025-10-10T01:03:58.7413502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7413916Z outputs = self.model( 2025-10-10T01:03:58.7414302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7414724Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7415105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7415498Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7415906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7416307Z return func(*args, **kwargs) 2025-10-10T01:03:58.7416703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7417138Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7417553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7417961Z return func(*args, **kwargs) 2025-10-10T01:03:58.7418397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7418849Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7419032Z 2025-10-10T01:03:58.7419146Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7419530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7419874Z res = mod(**inputs) 2025-10-10T01:03:58.7420249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7420654Z outputs = self.model( 2025-10-10T01:03:58.7421039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7421450Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7421823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7422219Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7422629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7423031Z return func(*args, **kwargs) 2025-10-10T01:03:58.7423430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7423865Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7424310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7424722Z return func(*args, **kwargs) 2025-10-10T01:03:58.7425131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7425628Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7425839Z 2025-10-10T01:03:58.7425959Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7426452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7426854Z res = mod(**inputs) 2025-10-10T01:03:58.7427267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7427678Z outputs = self.model( 2025-10-10T01:03:58.7428071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7428485Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7428873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7429276Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7429707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7430118Z return func(*args, **kwargs) 2025-10-10T01:03:58.7430516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7430959Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7431392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7431901Z return func(*args, **kwargs) 2025-10-10T01:03:58.7432307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7432738Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7432891Z 2025-10-10T01:03:58.7433015Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7433584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7433962Z res = mod(**inputs) 2025-10-10T01:03:58.7434348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7434736Z outputs = self.model( 2025-10-10T01:03:58.7435101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7435482Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7435843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7436215Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7436603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7436976Z return func(*args, **kwargs) 2025-10-10T01:03:58.7437355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7437771Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7438178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7438555Z return func(*args, **kwargs) 2025-10-10T01:03:58.7438920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7439330Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7439492Z 2025-10-10T01:03:58.7439598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7439961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7440289Z res = mod(**inputs) 2025-10-10T01:03:58.7440646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7441034Z outputs = self.model( 2025-10-10T01:03:58.7441396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7441821Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7442173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7442546Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7442936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7443313Z return func(*args, **kwargs) 2025-10-10T01:03:58.7443691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7444097Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7444547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7444929Z return func(*args, **kwargs) 2025-10-10T01:03:58.7445305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7445775Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7445967Z 2025-10-10T01:03:58.7446081Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7446467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7446815Z res = mod(**inputs) 2025-10-10T01:03:58.7447177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7447555Z outputs = self.model( 2025-10-10T01:03:58.7447942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7448350Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7448709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7449086Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7449464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7449843Z return func(*args, **kwargs) 2025-10-10T01:03:58.7450220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7450632Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7451041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7451417Z return func(*args, **kwargs) 2025-10-10T01:03:58.7451793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7452188Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7452332Z 2025-10-10T01:03:58.7452446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7452806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7453132Z res = mod(**inputs) 2025-10-10T01:03:58.7453493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7453879Z outputs = self.model( 2025-10-10T01:03:58.7454242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7454622Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7454987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7455363Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7455772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7456190Z return func(*args, **kwargs) 2025-10-10T01:03:58.7456589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7457057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7457254Z 2025-10-10T01:03:58.7457367Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7457734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7458075Z res = mod(**inputs) 2025-10-10T01:03:58.7458462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7458895Z outputs = self.model( 2025-10-10T01:03:58.7459289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7459706Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7460079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7460474Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7460886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7461303Z return func(*args, **kwargs) 2025-10-10T01:03:58.7461698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7462160Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7462623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7463009Z return self.act(input) 2025-10-10T01:03:58.7463134Z 2025-10-10T01:03:58.7463258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7463659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7464023Z res = mod(**inputs) 2025-10-10T01:03:58.7464430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7464859Z outputs = self.model( 2025-10-10T01:03:58.7465260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7465678Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7466067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7466564Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7466993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7467415Z return func(*args, **kwargs) 2025-10-10T01:03:58.7467835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7468260Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7468411Z 2025-10-10T01:03:58.7468533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7468929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7469279Z res = mod(**inputs) 2025-10-10T01:03:58.7469668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7470081Z outputs = self.model( 2025-10-10T01:03:58.7470471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7470883Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7471262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7471682Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7472156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7472569Z return func(*args, **kwargs) 2025-10-10T01:03:58.7472954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7473351Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7473499Z 2025-10-10T01:03:58.7473605Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7473990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7474321Z res = mod(**inputs) 2025-10-10T01:03:58.7474677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7475068Z outputs = self.model( 2025-10-10T01:03:58.7475452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7475867Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7476242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7476639Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7477057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7477442Z return func(*args, **kwargs) 2025-10-10T01:03:58.7477852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7478260Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7478668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7479050Z return func(*args, **kwargs) 2025-10-10T01:03:58.7479427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7479873Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7480054Z 2025-10-10T01:03:58.7480168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7480557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7480901Z res = mod(**inputs) 2025-10-10T01:03:58.7481289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7481688Z outputs = self.model( 2025-10-10T01:03:58.7482077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7482490Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7482868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7483270Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7483668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7484048Z return func(*args, **kwargs) 2025-10-10T01:03:58.7484425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7484842Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7485249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7485652Z return func(*args, **kwargs) 2025-10-10T01:03:58.7486074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7486499Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7486646Z 2025-10-10T01:03:58.7486768Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7487150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7487495Z res = mod(**inputs) 2025-10-10T01:03:58.7487876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7488284Z outputs = self.model( 2025-10-10T01:03:58.7488694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7489100Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7489470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7489858Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7490266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7490658Z return func(*args, **kwargs) 2025-10-10T01:03:58.7491056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7491493Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7491924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7492365Z return func(*args, **kwargs) 2025-10-10T01:03:58.7492754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7493204Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7493390Z 2025-10-10T01:03:58.7493503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7493892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7494238Z res = mod(**inputs) 2025-10-10T01:03:58.7494614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7495019Z outputs = self.model( 2025-10-10T01:03:58.7495403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7495826Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7496204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7496597Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7497007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7497418Z return func(*args, **kwargs) 2025-10-10T01:03:58.7497812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7498243Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7498669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7499071Z return func(*args, **kwargs) 2025-10-10T01:03:58.7499475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7499936Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7500126Z 2025-10-10T01:03:58.7500234Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7500598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7500977Z res = mod(**inputs) 2025-10-10T01:03:58.7501359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7501771Z outputs = self.model( 2025-10-10T01:03:58.7502159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7502574Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7502951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7503352Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7503774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7504173Z return func(*args, **kwargs) 2025-10-10T01:03:58.7504568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7505007Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7505436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7505827Z return func(*args, **kwargs) 2025-10-10T01:03:58.7506292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7506747Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7506907Z 2025-10-10T01:03:58.7507032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7507469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7507839Z res = mod(**inputs) 2025-10-10T01:03:58.7508226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7508642Z outputs = self.model( 2025-10-10T01:03:58.7509034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7509444Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7509830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7510227Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7510638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7511018Z return func(*args, **kwargs) 2025-10-10T01:03:58.7511398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7511820Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7512231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7512610Z return func(*args, **kwargs) 2025-10-10T01:03:58.7512982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7513399Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7513562Z 2025-10-10T01:03:58.7513669Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7514036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7514367Z res = mod(**inputs) 2025-10-10T01:03:58.7514729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7515118Z outputs = self.model( 2025-10-10T01:03:58.7515486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7515908Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7516285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7516690Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7517100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7517513Z return func(*args, **kwargs) 2025-10-10T01:03:58.7517910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7518022Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7518302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7518391Z return func(*args, **kwargs) 2025-10-10T01:03:58.7518662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7518812Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7518816Z 2025-10-10T01:03:58.7518927Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7519149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7519220Z res = mod(**inputs) 2025-10-10T01:03:58.7519489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7519590Z outputs = self.model( 2025-10-10T01:03:58.7519869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7519959Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7520198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7520288Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7520554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7520629Z return func(*args, **kwargs) 2025-10-10T01:03:58.7520901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7521006Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7521266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7521352Z return func(*args, **kwargs) 2025-10-10T01:03:58.7521621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7521718Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7521725Z 2025-10-10T01:03:58.7521838Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7522063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7522134Z res = mod(**inputs) 2025-10-10T01:03:58.7522401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7522485Z outputs = self.model( 2025-10-10T01:03:58.7522750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7522842Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7523084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7523172Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7523439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7523534Z return func(*args, **kwargs) 2025-10-10T01:03:58.7523807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7523940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7523944Z 2025-10-10T01:03:58.7524061Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7524275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7524344Z res = mod(**inputs) 2025-10-10T01:03:58.7524637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7524712Z outputs = self.model( 2025-10-10T01:03:58.7524981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7525061Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7525302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7525396Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7525663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7525748Z return func(*args, **kwargs) 2025-10-10T01:03:58.7526021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7526192Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7526437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7526515Z return self.act(input) 2025-10-10T01:03:58.7526519Z 2025-10-10T01:03:58.7526642Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7526859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7526940Z res = mod(**inputs) 2025-10-10T01:03:58.7527211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7527287Z outputs = self.model( 2025-10-10T01:03:58.7527565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7527649Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7527913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7528002Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7528260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7528347Z return func(*args, **kwargs) 2025-10-10T01:03:58.7528611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7528708Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7528712Z 2025-10-10T01:03:58.7528825Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7529036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7529115Z res = mod(**inputs) 2025-10-10T01:03:58.7529378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7529462Z outputs = self.model( 2025-10-10T01:03:58.7529729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7529817Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7530078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7530165Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7530430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7530507Z return func(*args, **kwargs) 2025-10-10T01:03:58.7530776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7530885Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7531163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7531250Z return func(*args, **kwargs) 2025-10-10T01:03:58.7531661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7531802Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7531806Z 2025-10-10T01:03:58.7531918Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7532141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7532212Z res = mod(**inputs) 2025-10-10T01:03:58.7532477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7532562Z outputs = self.model( 2025-10-10T01:03:58.7532874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7533001Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7533241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7533329Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7533598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7533673Z return func(*args, **kwargs) 2025-10-10T01:03:58.7533951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7534060Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7534319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7534405Z return func(*args, **kwargs) 2025-10-10T01:03:58.7534678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7534777Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7534781Z 2025-10-10T01:03:58.7534894Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7535115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7535185Z res = mod(**inputs) 2025-10-10T01:03:58.7535453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7535534Z outputs = self.model( 2025-10-10T01:03:58.7535800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7535887Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7536128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7536216Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7536484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7536588Z return func(*args, **kwargs) 2025-10-10T01:03:58.7536856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7536968Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7537210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7537289Z return func(*args, **kwargs) 2025-10-10T01:03:58.7537537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7537660Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7537666Z 2025-10-10T01:03:58.7537795Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7538003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7538069Z res = mod(**inputs) 2025-10-10T01:03:58.7538319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7538398Z outputs = self.model( 2025-10-10T01:03:58.7538644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7538726Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7538949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7539029Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7539296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7539385Z return func(*args, **kwargs) 2025-10-10T01:03:58.7539640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7539740Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7539989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7540059Z return func(*args, **kwargs) 2025-10-10T01:03:58.7540310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7540457Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7540461Z 2025-10-10T01:03:58.7540565Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7540772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7540846Z res = mod(**inputs) 2025-10-10T01:03:58.7541110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7541192Z outputs = self.model( 2025-10-10T01:03:58.7541458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7541542Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7541780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7541864Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7542128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7542203Z return func(*args, **kwargs) 2025-10-10T01:03:58.7542474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7542581Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7542844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7542940Z return func(*args, **kwargs) 2025-10-10T01:03:58.7543212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7543315Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7543319Z 2025-10-10T01:03:58.7543434Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7543652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7543721Z res = mod(**inputs) 2025-10-10T01:03:58.7543986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7544085Z outputs = self.model( 2025-10-10T01:03:58.7544347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7544435Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7544673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7544766Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7545030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7545105Z return func(*args, **kwargs) 2025-10-10T01:03:58.7545374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7545477Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7545802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7545882Z return func(*args, **kwargs) 2025-10-10T01:03:58.7546153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7546333Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7546340Z 2025-10-10T01:03:58.7546457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7546683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7546755Z res = mod(**inputs) 2025-10-10T01:03:58.7547025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7547109Z outputs = self.model( 2025-10-10T01:03:58.7547382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7547475Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7547724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7547814Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7548057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7548128Z return func(*args, **kwargs) 2025-10-10T01:03:58.7548382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7548484Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7548737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7548811Z return func(*args, **kwargs) 2025-10-10T01:03:58.7549078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7549227Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7549231Z 2025-10-10T01:03:58.7549365Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7549588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7549660Z res = mod(**inputs) 2025-10-10T01:03:58.7549943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7550029Z outputs = self.model( 2025-10-10T01:03:58.7550282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7550365Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7550589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7550695Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7550946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7551020Z return func(*args, **kwargs) 2025-10-10T01:03:58.7551279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7551379Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7551632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7551703Z return func(*args, **kwargs) 2025-10-10T01:03:58.7551957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7552083Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7552087Z 2025-10-10T01:03:58.7552195Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7552402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7552470Z res = mod(**inputs) 2025-10-10T01:03:58.7552725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7552795Z outputs = self.model( 2025-10-10T01:03:58.7553043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7553126Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7553349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7553437Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7553683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7553757Z return func(*args, **kwargs) 2025-10-10T01:03:58.7554012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7554135Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7554139Z 2025-10-10T01:03:58.7554250Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7554447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7554522Z res = mod(**inputs) 2025-10-10T01:03:58.7554769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7554838Z outputs = self.model( 2025-10-10T01:03:58.7555095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7555173Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7555402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7555484Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7555767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7555846Z return func(*args, **kwargs) 2025-10-10T01:03:58.7556099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7556226Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7556445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7556516Z return self.act(input) 2025-10-10T01:03:58.7556521Z 2025-10-10T01:03:58.7556632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7556850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7556925Z res = mod(**inputs) 2025-10-10T01:03:58.7557174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7557252Z outputs = self.model( 2025-10-10T01:03:58.7557502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7557576Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7557807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7557888Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7558166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7558254Z return func(*args, **kwargs) 2025-10-10T01:03:58.7558506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7558598Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7558604Z 2025-10-10T01:03:58.7558710Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7558917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7558984Z res = mod(**inputs) 2025-10-10T01:03:58.7559234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7559312Z outputs = self.model( 2025-10-10T01:03:58.7559577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7559663Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7559909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7560010Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7560257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7560330Z return func(*args, **kwargs) 2025-10-10T01:03:58.7560588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7560672Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7560676Z 2025-10-10T01:03:58.7560790Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7560989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7561056Z res = mod(**inputs) 2025-10-10T01:03:58.7561331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7561407Z outputs = self.model( 2025-10-10T01:03:58.7561682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7561781Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7562022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7562117Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7562385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7562469Z return func(*args, **kwargs) 2025-10-10T01:03:58.7562734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7562852Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7563147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7563220Z return func(*args, **kwargs) 2025-10-10T01:03:58.7563473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7563591Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7563595Z 2025-10-10T01:03:58.7563718Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7563914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7563979Z res = mod(**inputs) 2025-10-10T01:03:58.7564227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7564294Z outputs = self.model( 2025-10-10T01:03:58.7564581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7564657Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7564887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7564969Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7565215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7565295Z return func(*args, **kwargs) 2025-10-10T01:03:58.7565544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7565650Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7565894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7565968Z return func(*args, **kwargs) 2025-10-10T01:03:58.7566225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7566308Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7566313Z 2025-10-10T01:03:58.7566426Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7566627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7566693Z res = mod(**inputs) 2025-10-10T01:03:58.7566950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7567020Z outputs = self.model( 2025-10-10T01:03:58.7567276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7567350Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7567585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7567667Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7567911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7568667Z return func(*args, **kwargs) 2025-10-10T01:03:58.7568922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7569030Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7569274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7569343Z return func(*args, **kwargs) 2025-10-10T01:03:58.7569604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7569721Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7569747Z 2025-10-10T01:03:58.7569861Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7570062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7570137Z res = mod(**inputs) 2025-10-10T01:03:58.7570387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7570459Z outputs = self.model( 2025-10-10T01:03:58.7570729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7570808Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7571053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7571137Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7571435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7571518Z return func(*args, **kwargs) 2025-10-10T01:03:58.7571781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7571893Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7572158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7572233Z return func(*args, **kwargs) 2025-10-10T01:03:58.7572503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7572649Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7572653Z 2025-10-10T01:03:58.7572772Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7572988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7573064Z res = mod(**inputs) 2025-10-10T01:03:58.7573330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7573406Z outputs = self.model( 2025-10-10T01:03:58.7573677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7573755Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7573999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7574083Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7574351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7574437Z return func(*args, **kwargs) 2025-10-10T01:03:58.7574705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7574816Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7575071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7575165Z return func(*args, **kwargs) 2025-10-10T01:03:58.7575437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7575528Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7575531Z 2025-10-10T01:03:58.7575643Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7575843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7575916Z res = mod(**inputs) 2025-10-10T01:03:58.7576182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7576255Z outputs = self.model( 2025-10-10T01:03:58.7576513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7576589Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7576825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7576906Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7577150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7577229Z return func(*args, **kwargs) 2025-10-10T01:03:58.7577477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7577617Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7577868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7577946Z return func(*args, **kwargs) 2025-10-10T01:03:58.7578204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7578305Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7578309Z 2025-10-10T01:03:58.7578422Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7578623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7578696Z res = mod(**inputs) 2025-10-10T01:03:58.7578952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7579023Z outputs = self.model( 2025-10-10T01:03:58.7579287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7579363Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7579599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7579683Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7579927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7580006Z return func(*args, **kwargs) 2025-10-10T01:03:58.7580274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7580388Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7580650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7580735Z return func(*args, **kwargs) 2025-10-10T01:03:58.7581003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7581142Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7581164Z 2025-10-10T01:03:58.7581283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7581492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7581570Z res = mod(**inputs) 2025-10-10T01:03:58.7581836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7581910Z outputs = self.model( 2025-10-10T01:03:58.7582179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7582257Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7582527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7582615Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7582878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7582955Z return func(*args, **kwargs) 2025-10-10T01:03:58.7583219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7583332Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7583588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7583670Z return func(*args, **kwargs) 2025-10-10T01:03:58.7583952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7584062Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7584066Z 2025-10-10T01:03:58.7584187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7584399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7584481Z res = mod(**inputs) 2025-10-10T01:03:58.7584753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7584830Z outputs = self.model( 2025-10-10T01:03:58.7585108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7585190Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7585440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7585529Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7585807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7585884Z return func(*args, **kwargs) 2025-10-10T01:03:58.7586159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7586382Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7586389Z 2025-10-10T01:03:58.7586507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7586740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7586812Z res = mod(**inputs) 2025-10-10T01:03:58.7587085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7587178Z outputs = self.model( 2025-10-10T01:03:58.7587446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7587534Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7587783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7587893Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7588145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7588216Z return func(*args, **kwargs) 2025-10-10T01:03:58.7588477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7588599Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7588831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7588905Z return self.act(input) 2025-10-10T01:03:58.7588909Z 2025-10-10T01:03:58.7589030Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7589237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7589303Z res = mod(**inputs) 2025-10-10T01:03:58.7589557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7589627Z outputs = self.model( 2025-10-10T01:03:58.7589871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7589955Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7590177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7590266Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7590526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7590623Z return func(*args, **kwargs) 2025-10-10T01:03:58.7590871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7590958Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7590961Z 2025-10-10T01:03:58.7591075Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7591276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7591348Z res = mod(**inputs) 2025-10-10T01:03:58.7591599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7591668Z outputs = self.model( 2025-10-10T01:03:58.7591925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7592005Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7592238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7592319Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7592565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7592643Z return func(*args, **kwargs) 2025-10-10T01:03:58.7592890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7592997Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7593243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7593324Z return func(*args, **kwargs) 2025-10-10T01:03:58.7593584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7593693Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7593697Z 2025-10-10T01:03:58.7593803Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7594009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7594078Z res = mod(**inputs) 2025-10-10T01:03:58.7594320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7594387Z outputs = self.model( 2025-10-10T01:03:58.7594641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7594714Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7594938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7595034Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7595282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7595353Z return func(*args, **kwargs) 2025-10-10T01:03:58.7595604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7595710Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7595963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7596038Z return func(*args, **kwargs) 2025-10-10T01:03:58.7596280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7596360Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7596379Z 2025-10-10T01:03:58.7596505Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7596700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7596771Z res = mod(**inputs) 2025-10-10T01:03:58.7597013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7597092Z outputs = self.model( 2025-10-10T01:03:58.7597335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7597406Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7597622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7597698Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7597936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7598007Z return func(*args, **kwargs) 2025-10-10T01:03:58.7598244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7598348Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7598584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7598662Z return func(*args, **kwargs) 2025-10-10T01:03:58.7598904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7599015Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7599018Z 2025-10-10T01:03:58.7599127Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7599321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7599396Z res = mod(**inputs) 2025-10-10T01:03:58.7599639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7599711Z outputs = self.model( 2025-10-10T01:03:58.7599970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7600043Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7600271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7600351Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7600602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7600672Z return func(*args, **kwargs) 2025-10-10T01:03:58.7600921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7601049Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7601292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7601371Z return func(*args, **kwargs) 2025-10-10T01:03:58.7601631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7601776Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7601788Z 2025-10-10T01:03:58.7601899Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7602109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7602185Z res = mod(**inputs) 2025-10-10T01:03:58.7602466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7602567Z outputs = self.model( 2025-10-10T01:03:58.7602837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7602911Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7603148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7603227Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7603478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7603548Z return func(*args, **kwargs) 2025-10-10T01:03:58.7603795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7603901Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7604148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7604226Z return func(*args, **kwargs) 2025-10-10T01:03:58.7604474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7604564Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7604577Z 2025-10-10T01:03:58.7604682Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7604881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7604956Z res = mod(**inputs) 2025-10-10T01:03:58.7605206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7605283Z outputs = self.model( 2025-10-10T01:03:58.7605535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7605613Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7605847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7605927Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7606200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7606274Z return func(*args, **kwargs) 2025-10-10T01:03:58.7606531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7606640Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7606889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7606968Z return func(*args, **kwargs) 2025-10-10T01:03:58.7607244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7607351Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7607355Z 2025-10-10T01:03:58.7607458Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7607661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7607739Z res = mod(**inputs) 2025-10-10T01:03:58.7608015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7608096Z outputs = self.model( 2025-10-10T01:03:58.7608372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7608450Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7608712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7608818Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7609083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7609159Z return func(*args, **kwargs) 2025-10-10T01:03:58.7609429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7609550Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7609793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7609870Z return func(*args, **kwargs) 2025-10-10T01:03:58.7610122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7610268Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7610275Z 2025-10-10T01:03:58.7610388Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7610599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7610677Z res = mod(**inputs) 2025-10-10T01:03:58.7610942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7611024Z outputs = self.model( 2025-10-10T01:03:58.7611345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7611426Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7611672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7611758Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7612028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7612108Z return func(*args, **kwargs) 2025-10-10T01:03:58.7612378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7612506Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7612769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7612852Z return func(*args, **kwargs) 2025-10-10T01:03:58.7613124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7613219Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7613222Z 2025-10-10T01:03:58.7613334Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7613557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7613653Z res = mod(**inputs) 2025-10-10T01:03:58.7613930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7614011Z outputs = self.model( 2025-10-10T01:03:58.7614276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7614356Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7614601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7614686Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7614955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7615031Z return func(*args, **kwargs) 2025-10-10T01:03:58.7615319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7615464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7615468Z 2025-10-10T01:03:58.7615578Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7615797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7615867Z res = mod(**inputs) 2025-10-10T01:03:58.7616140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7616212Z outputs = self.model( 2025-10-10T01:03:58.7616477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7616562Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7616856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7616957Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7617215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7617295Z return func(*args, **kwargs) 2025-10-10T01:03:58.7617562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7617687Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7617925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7618002Z return self.act(input) 2025-10-10T01:03:58.7618006Z 2025-10-10T01:03:58.7618122Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7618335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7618408Z res = mod(**inputs) 2025-10-10T01:03:58.7618680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7618755Z outputs = self.model( 2025-10-10T01:03:58.7619023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7619121Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7619360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7619454Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7619710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7619792Z return func(*args, **kwargs) 2025-10-10T01:03:58.7620053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7620151Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7620172Z 2025-10-10T01:03:58.7620287Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7620499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7620579Z res = mod(**inputs) 2025-10-10T01:03:58.7620844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7620924Z outputs = self.model( 2025-10-10T01:03:58.7621189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7621269Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7621515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7621601Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7621898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7621974Z return func(*args, **kwargs) 2025-10-10T01:03:58.7622243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7622342Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7622346Z 2025-10-10T01:03:58.7622457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7622677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7622748Z res = mod(**inputs) 2025-10-10T01:03:58.7623022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7623095Z outputs = self.model( 2025-10-10T01:03:58.7623363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7623452Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7623693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7623787Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7624047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7624122Z return func(*args, **kwargs) 2025-10-10T01:03:58.7624394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7624501Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7624771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7624846Z return func(*args, **kwargs) 2025-10-10T01:03:58.7625114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7625243Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7625247Z 2025-10-10T01:03:58.7625376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7625597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7625666Z res = mod(**inputs) 2025-10-10T01:03:58.7625936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7626010Z outputs = self.model( 2025-10-10T01:03:58.7626349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7626443Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7626715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7626817Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7627084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7627166Z return func(*args, **kwargs) 2025-10-10T01:03:58.7627446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7627557Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7627848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7627924Z return func(*args, **kwargs) 2025-10-10T01:03:58.7628199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7628336Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7628341Z 2025-10-10T01:03:58.7628457Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7628678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7628751Z res = mod(**inputs) 2025-10-10T01:03:58.7629022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7629095Z outputs = self.model( 2025-10-10T01:03:58.7629360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7629448Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7629684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7629777Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7630049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7630126Z return func(*args, **kwargs) 2025-10-10T01:03:58.7630407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7630517Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7630792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7630868Z return func(*args, **kwargs) 2025-10-10T01:03:58.7631145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7631270Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7631274Z 2025-10-10T01:03:58.7631388Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7631802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7631885Z res = mod(**inputs) 2025-10-10T01:03:58.7632167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7632290Z outputs = self.model( 2025-10-10T01:03:58.7632560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7632648Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7632892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7632985Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7633248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7633330Z return func(*args, **kwargs) 2025-10-10T01:03:58.7633641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7633750Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7634025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7634107Z return func(*args, **kwargs) 2025-10-10T01:03:58.7634390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7634539Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7634544Z 2025-10-10T01:03:58.7634659Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7634886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7634958Z res = mod(**inputs) 2025-10-10T01:03:58.7635269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7635378Z outputs = self.model( 2025-10-10T01:03:58.7635653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7635752Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7635978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7636068Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7636313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7636392Z return func(*args, **kwargs) 2025-10-10T01:03:58.7636642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7636744Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7637001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7637073Z return func(*args, **kwargs) 2025-10-10T01:03:58.7637330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7637422Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7637426Z 2025-10-10T01:03:58.7637531Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7637740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7637807Z res = mod(**inputs) 2025-10-10T01:03:58.7638064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7638133Z outputs = self.model( 2025-10-10T01:03:58.7638391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7638467Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7638694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7638800Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7639043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7639123Z return func(*args, **kwargs) 2025-10-10T01:03:58.7639370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7639467Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7639717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7639790Z return func(*args, **kwargs) 2025-10-10T01:03:58.7640062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7640164Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7640169Z 2025-10-10T01:03:58.7640280Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7640480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7640545Z res = mod(**inputs) 2025-10-10T01:03:58.7640800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7640870Z outputs = self.model( 2025-10-10T01:03:58.7641124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7641199Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7641457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7641547Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7641789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7641869Z return func(*args, **kwargs) 2025-10-10T01:03:58.7642119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7642217Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7642468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7642540Z return func(*args, **kwargs) 2025-10-10T01:03:58.7642796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7642931Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7642936Z 2025-10-10T01:03:58.7643051Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7643248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7643319Z res = mod(**inputs) 2025-10-10T01:03:58.7643578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7643648Z outputs = self.model( 2025-10-10T01:03:58.7643904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7643980Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7644203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7644292Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7644538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7644615Z return func(*args, **kwargs) 2025-10-10T01:03:58.7644866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7644983Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7645234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7645306Z return func(*args, **kwargs) 2025-10-10T01:03:58.7645566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7645651Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7645654Z 2025-10-10T01:03:58.7645768Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7645986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7646055Z res = mod(**inputs) 2025-10-10T01:03:58.7646313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7646385Z outputs = self.model( 2025-10-10T01:03:58.7646641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7646715Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7646945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7647031Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7647281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7647360Z return func(*args, **kwargs) 2025-10-10T01:03:58.7647640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7647761Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7647772Z 2025-10-10T01:03:58.7647877Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7648075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7648151Z res = mod(**inputs) 2025-10-10T01:03:58.7648399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7648477Z outputs = self.model( 2025-10-10T01:03:58.7648723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7648797Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7649028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7649110Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7649359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7649432Z return func(*args, **kwargs) 2025-10-10T01:03:58.7649680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7649803Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7650019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7650099Z return self.act(input) 2025-10-10T01:03:58.7650102Z 2025-10-10T01:03:58.7650207Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7650414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7650481Z res = mod(**inputs) 2025-10-10T01:03:58.7650741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7650822Z outputs = self.model( 2025-10-10T01:03:58.7651102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7651192Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7651427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7651513Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7651794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7651870Z return func(*args, **kwargs) 2025-10-10T01:03:58.7652144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7652256Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7652260Z 2025-10-10T01:03:58.7652371Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7652588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7652659Z res = mod(**inputs) 2025-10-10T01:03:58.7652928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7653002Z outputs = self.model( 2025-10-10T01:03:58.7653274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7653353Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7653589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7653747Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7654012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7654095Z return func(*args, **kwargs) 2025-10-10T01:03:58.7654362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7654470Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7654745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7654819Z return func(*args, **kwargs) 2025-10-10T01:03:58.7655094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7655220Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7655225Z 2025-10-10T01:03:58.7655344Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7655558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7655629Z res = mod(**inputs) 2025-10-10T01:03:58.7655906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7655981Z outputs = self.model( 2025-10-10T01:03:58.7656254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7656333Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7656573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7656667Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7656938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7657021Z return func(*args, **kwargs) 2025-10-10T01:03:58.7657290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7657394Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7657688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7657765Z return func(*args, **kwargs) 2025-10-10T01:03:58.7658037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7658125Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7658129Z 2025-10-10T01:03:58.7658247Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7658471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7658544Z res = mod(**inputs) 2025-10-10T01:03:58.7658837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7658913Z outputs = self.model( 2025-10-10T01:03:58.7659185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7659266Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7659502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7659595Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7659852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7659935Z return func(*args, **kwargs) 2025-10-10T01:03:58.7660216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7660342Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7660608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7660686Z return func(*args, **kwargs) 2025-10-10T01:03:58.7660960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7661074Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7661077Z 2025-10-10T01:03:58.7661189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7661389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7661456Z res = mod(**inputs) 2025-10-10T01:03:58.7661711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7661785Z outputs = self.model( 2025-10-10T01:03:58.7662041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7662115Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7662343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7662430Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7662672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7662750Z return func(*args, **kwargs) 2025-10-10T01:03:58.7662998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7663103Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7663349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7663423Z return func(*args, **kwargs) 2025-10-10T01:03:58.7663678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7663833Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7663836Z 2025-10-10T01:03:58.7663948Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7664148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7664214Z res = mod(**inputs) 2025-10-10T01:03:58.7664470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7664540Z outputs = self.model( 2025-10-10T01:03:58.7664794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7664872Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7665114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7665205Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7665453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7665535Z return func(*args, **kwargs) 2025-10-10T01:03:58.7665786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7665893Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7666149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7666305Z return func(*args, **kwargs) 2025-10-10T01:03:58.7666686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7666801Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7666805Z 2025-10-10T01:03:58.7666924Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7667137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7667207Z res = mod(**inputs) 2025-10-10T01:03:58.7667479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7667555Z outputs = self.model( 2025-10-10T01:03:58.7667828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7667908Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7668153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7668246Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7668503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7668588Z return func(*args, **kwargs) 2025-10-10T01:03:58.7668854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7668966Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7669222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7669298Z return func(*args, **kwargs) 2025-10-10T01:03:58.7669570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7669673Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7669679Z 2025-10-10T01:03:58.7669797Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7670007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7670077Z res = mod(**inputs) 2025-10-10T01:03:58.7670367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7670442Z outputs = self.model( 2025-10-10T01:03:58.7670728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7670807Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7671050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7671135Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7671405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7671505Z return func(*args, **kwargs) 2025-10-10T01:03:58.7671767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7671880Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7672146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7672221Z return func(*args, **kwargs) 2025-10-10T01:03:58.7672502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7672640Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7672644Z 2025-10-10T01:03:58.7672764Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7673000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7673091Z res = mod(**inputs) 2025-10-10T01:03:58.7673350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7673419Z outputs = self.model( 2025-10-10T01:03:58.7673683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7673758Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7673989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7674071Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7674340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7674426Z return func(*args, **kwargs) 2025-10-10T01:03:58.7674706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7674822Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7675083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7675160Z return func(*args, **kwargs) 2025-10-10T01:03:58.7675444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7675534Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7675538Z 2025-10-10T01:03:58.7675657Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7675878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7675956Z res = mod(**inputs) 2025-10-10T01:03:58.7676235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7676311Z outputs = self.model( 2025-10-10T01:03:58.7676586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7676664Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7676927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7677012Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7677278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7677361Z return func(*args, **kwargs) 2025-10-10T01:03:58.7677633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7677770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7677774Z 2025-10-10T01:03:58.7677886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7678132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7678204Z res = mod(**inputs) 2025-10-10T01:03:58.7678524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7678607Z outputs = self.model( 2025-10-10T01:03:58.7678882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7678968Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7679205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7679290Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7679572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7679683Z return func(*args, **kwargs) 2025-10-10T01:03:58.7679959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7680084Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7680318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7680400Z return self.act(input) 2025-10-10T01:03:58.7680404Z 2025-10-10T01:03:58.7680513Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7680734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7680803Z res = mod(**inputs) 2025-10-10T01:03:58.7681074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7681147Z outputs = self.model( 2025-10-10T01:03:58.7681414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7681502Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7681736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7681832Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7682092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7682169Z return func(*args, **kwargs) 2025-10-10T01:03:58.7682441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7682528Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7682531Z 2025-10-10T01:03:58.7682649Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7682865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7682936Z res = mod(**inputs) 2025-10-10T01:03:58.7683205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7683299Z outputs = self.model( 2025-10-10T01:03:58.7683569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7683648Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7683892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7683977Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7684236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7684320Z return func(*args, **kwargs) 2025-10-10T01:03:58.7684605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7684702Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7684706Z 2025-10-10T01:03:58.7684818Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7685032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7685109Z res = mod(**inputs) 2025-10-10T01:03:58.7685371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7685450Z outputs = self.model( 2025-10-10T01:03:58.7685712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7685790Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7686053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7686156Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7686418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7686496Z return func(*args, **kwargs) 2025-10-10T01:03:58.7686765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7686872Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7687129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7687212Z return func(*args, **kwargs) 2025-10-10T01:03:58.7687475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7687602Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7687608Z 2025-10-10T01:03:58.7687721Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7687932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7688011Z res = mod(**inputs) 2025-10-10T01:03:58.7688274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7688355Z outputs = self.model( 2025-10-10T01:03:58.7688617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7688702Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7688938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7689025Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7689295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7689377Z return func(*args, **kwargs) 2025-10-10T01:03:58.7689654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7689783Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7690053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7690138Z return func(*args, **kwargs) 2025-10-10T01:03:58.7690417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7690514Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7690518Z 2025-10-10T01:03:58.7690632Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7691327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7691417Z res = mod(**inputs) 2025-10-10T01:03:58.7691684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7691768Z outputs = self.model( 2025-10-10T01:03:58.7692031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7692118Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7692353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7692437Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7692702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7692774Z return func(*args, **kwargs) 2025-10-10T01:03:58.7693063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7693186Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7693442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7693525Z return func(*args, **kwargs) 2025-10-10T01:03:58.7693791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7693918Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7693922Z 2025-10-10T01:03:58.7694032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7694250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7694319Z res = mod(**inputs) 2025-10-10T01:03:58.7694593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7694678Z outputs = self.model( 2025-10-10T01:03:58.7694950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7695051Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7695288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7695373Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7695642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7695720Z return func(*args, **kwargs) 2025-10-10T01:03:58.7696000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7696106Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7696381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7696458Z return func(*args, **kwargs) 2025-10-10T01:03:58.7696732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7696910Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7696914Z 2025-10-10T01:03:58.7697028Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7697252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7697325Z res = mod(**inputs) 2025-10-10T01:03:58.7697600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7697684Z outputs = self.model( 2025-10-10T01:03:58.7697959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7698067Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7698318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7698408Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7698688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7698766Z return func(*args, **kwargs) 2025-10-10T01:03:58.7699048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7699157Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7699435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7699511Z return func(*args, **kwargs) 2025-10-10T01:03:58.7699823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7699932Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7699936Z 2025-10-10T01:03:58.7700052Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7700279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7700351Z res = mod(**inputs) 2025-10-10T01:03:58.7700624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7700708Z outputs = self.model( 2025-10-10T01:03:58.7700981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7701070Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7701319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7701417Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7701682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7701760Z return func(*args, **kwargs) 2025-10-10T01:03:58.7702041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7702150Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7702423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7702498Z return func(*args, **kwargs) 2025-10-10T01:03:58.7702769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7702888Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7702892Z 2025-10-10T01:03:58.7703006Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7703231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7703332Z res = mod(**inputs) 2025-10-10T01:03:58.7703601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7703682Z outputs = self.model( 2025-10-10T01:03:58.7703952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7704041Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7704287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7704382Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7704667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7704745Z return func(*args, **kwargs) 2025-10-10T01:03:58.7705020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7705129Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7705400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7705479Z return func(*args, **kwargs) 2025-10-10T01:03:58.7705750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7705901Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7705905Z 2025-10-10T01:03:58.7706019Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7706519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7706605Z res = mod(**inputs) 2025-10-10T01:03:58.7706895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7706974Z outputs = self.model( 2025-10-10T01:03:58.7707258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7707350Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7707600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7707698Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7708003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7708081Z return func(*args, **kwargs) 2025-10-10T01:03:58.7708373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7708482Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7708763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7708840Z return func(*args, **kwargs) 2025-10-10T01:03:58.7709120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7709219Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7709222Z 2025-10-10T01:03:58.7709336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7709564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7709635Z res = mod(**inputs) 2025-10-10T01:03:58.7709923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7710000Z outputs = self.model( 2025-10-10T01:03:58.7710277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7710386Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7710628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7710723Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7710987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7711062Z return func(*args, **kwargs) 2025-10-10T01:03:58.7711340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7711472Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7711479Z 2025-10-10T01:03:58.7711617Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7711837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7711913Z res = mod(**inputs) 2025-10-10T01:03:58.7712202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7712280Z outputs = self.model( 2025-10-10T01:03:58.7712575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7712656Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7712905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7712994Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7713283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7713392Z return func(*args, **kwargs) 2025-10-10T01:03:58.7713663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7713804Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7714042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7714131Z return self.act(input) 2025-10-10T01:03:58.7714135Z 2025-10-10T01:03:58.7714252Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7714473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7714551Z res = mod(**inputs) 2025-10-10T01:03:58.7714825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7714910Z outputs = self.model( 2025-10-10T01:03:58.7715196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7715275Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7715521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7715610Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7715888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7715962Z return func(*args, **kwargs) 2025-10-10T01:03:58.7716238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7716333Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7716337Z 2025-10-10T01:03:58.7716452Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7716675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7716747Z res = mod(**inputs) 2025-10-10T01:03:58.7717030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7717144Z outputs = self.model( 2025-10-10T01:03:58.7717442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7717532Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7717781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7717875Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7718157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7718238Z return func(*args, **kwargs) 2025-10-10T01:03:58.7718540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7718651Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7718928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7719005Z return func(*args, **kwargs) 2025-10-10T01:03:58.7719291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7719424Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7719428Z 2025-10-10T01:03:58.7719544Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7719780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7719869Z res = mod(**inputs) 2025-10-10T01:03:58.7720167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7720245Z outputs = self.model( 2025-10-10T01:03:58.7720519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7720608Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7720846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7720939Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7721208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7721283Z return func(*args, **kwargs) 2025-10-10T01:03:58.7721552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7721663Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7721929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7722002Z return func(*args, **kwargs) 2025-10-10T01:03:58.7722267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7722363Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7722367Z 2025-10-10T01:03:58.7722478Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7722698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7722767Z res = mod(**inputs) 2025-10-10T01:03:58.7723040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7723114Z outputs = self.model( 2025-10-10T01:03:58.7723378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7723465Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7723702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7723813Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7724069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7724141Z return func(*args, **kwargs) 2025-10-10T01:03:58.7724411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7724516Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7724780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7724872Z return func(*args, **kwargs) 2025-10-10T01:03:58.7725135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7725262Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7725267Z 2025-10-10T01:03:58.7725378Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7725605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7725677Z res = mod(**inputs) 2025-10-10T01:03:58.7725953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7726029Z outputs = self.model( 2025-10-10T01:03:58.7726298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7726427Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7726672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7726768Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7727033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7727110Z return func(*args, **kwargs) 2025-10-10T01:03:58.7727389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7727496Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7727767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7727842Z return func(*args, **kwargs) 2025-10-10T01:03:58.7728120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7728271Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7728275Z 2025-10-10T01:03:58.7728390Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7728624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7728694Z res = mod(**inputs) 2025-10-10T01:03:58.7728965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7729038Z outputs = self.model( 2025-10-10T01:03:58.7729298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7729386Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7729623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7729720Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7729985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7730063Z return func(*args, **kwargs) 2025-10-10T01:03:58.7730359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7730467Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7730739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7730816Z return func(*args, **kwargs) 2025-10-10T01:03:58.7731096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7731193Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7731197Z 2025-10-10T01:03:58.7731314Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7731730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7731811Z res = mod(**inputs) 2025-10-10T01:03:58.7732091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7732170Z outputs = self.model( 2025-10-10T01:03:58.7732450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7732542Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7732779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7732874Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7733147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7733281Z return func(*args, **kwargs) 2025-10-10T01:03:58.7733605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7733713Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7733992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7734069Z return func(*args, **kwargs) 2025-10-10T01:03:58.7734394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7734500Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7734504Z 2025-10-10T01:03:58.7734618Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7734842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7734919Z res = mod(**inputs) 2025-10-10T01:03:58.7735205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7735281Z outputs = self.model( 2025-10-10T01:03:58.7735580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7735670Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7735913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7736007Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7736271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7736355Z return func(*args, **kwargs) 2025-10-10T01:03:58.7736637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7736746Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7737016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7737123Z return func(*args, **kwargs) 2025-10-10T01:03:58.7737416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7737559Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7737563Z 2025-10-10T01:03:58.7737677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7737903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7737976Z res = mod(**inputs) 2025-10-10T01:03:58.7738266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7738346Z outputs = self.model( 2025-10-10T01:03:58.7738664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7738748Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7738997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7739095Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7739371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7739457Z return func(*args, **kwargs) 2025-10-10T01:03:58.7739745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7739852Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7740151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7740256Z return func(*args, **kwargs) 2025-10-10T01:03:58.7740547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7740642Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7740646Z 2025-10-10T01:03:58.7740759Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7740984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7741055Z res = mod(**inputs) 2025-10-10T01:03:58.7741340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7741416Z outputs = self.model( 2025-10-10T01:03:58.7741707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7741792Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7742040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7742137Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7742419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7742502Z return func(*args, **kwargs) 2025-10-10T01:03:58.7742787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7742920Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7742924Z 2025-10-10T01:03:58.7743043Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7743271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7743353Z res = mod(**inputs) 2025-10-10T01:03:58.7743629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7743704Z outputs = self.model( 2025-10-10T01:03:58.7743984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7744083Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7744337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7744424Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7744699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7744776Z return func(*args, **kwargs) 2025-10-10T01:03:58.7745052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7745210Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7745451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7745537Z return self.act(input) 2025-10-10T01:03:58.7745543Z 2025-10-10T01:03:58.7745657Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7745873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7745953Z res = mod(**inputs) 2025-10-10T01:03:58.7746281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7746377Z outputs = self.model( 2025-10-10T01:03:58.7746648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7746738Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7747022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7747112Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7747386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7747466Z return func(*args, **kwargs) 2025-10-10T01:03:58.7747747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7747839Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7747844Z 2025-10-10T01:03:58.7747959Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7748185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7748257Z res = mod(**inputs) 2025-10-10T01:03:58.7748541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7748622Z outputs = self.model( 2025-10-10T01:03:58.7748894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7748997Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7749236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7749331Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7749591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7749671Z return func(*args, **kwargs) 2025-10-10T01:03:58.7749936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7750024Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7750029Z 2025-10-10T01:03:58.7750151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7750361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7750438Z res = mod(**inputs) 2025-10-10T01:03:58.7750725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7750798Z outputs = self.model( 2025-10-10T01:03:58.7751071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7751150Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7751396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7751480Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7751743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7751843Z return func(*args, **kwargs) 2025-10-10T01:03:58.7752108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7752222Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7752479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7752563Z return func(*args, **kwargs) 2025-10-10T01:03:58.7752825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7752945Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7752948Z 2025-10-10T01:03:58.7753068Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7753304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7753398Z res = mod(**inputs) 2025-10-10T01:03:58.7753663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7753737Z outputs = self.model( 2025-10-10T01:03:58.7754009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7754088Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7754330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7754416Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7754680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7754755Z return func(*args, **kwargs) 2025-10-10T01:03:58.7755022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7755139Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7755396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7755480Z return func(*args, **kwargs) 2025-10-10T01:03:58.7755744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7755832Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7755836Z 2025-10-10T01:03:58.7755954Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7756164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7756240Z res = mod(**inputs) 2025-10-10T01:03:58.7756505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7756580Z outputs = self.model( 2025-10-10T01:03:58.7756851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7756929Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7757196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7757281Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7757548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7757622Z return func(*args, **kwargs) 2025-10-10T01:03:58.7757891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7758001Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7758283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7758369Z return func(*args, **kwargs) 2025-10-10T01:03:58.7758637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7758760Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7758764Z 2025-10-10T01:03:58.7758882Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7759090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7759167Z res = mod(**inputs) 2025-10-10T01:03:58.7759432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7759515Z outputs = self.model( 2025-10-10T01:03:58.7759813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7759912Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7760164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7760249Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7760524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7760600Z return func(*args, **kwargs) 2025-10-10T01:03:58.7760872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7760985Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7761248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7761330Z return func(*args, **kwargs) 2025-10-10T01:03:58.7761606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7761751Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7761763Z 2025-10-10T01:03:58.7761878Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7762095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7762174Z res = mod(**inputs) 2025-10-10T01:03:58.7762449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7762530Z outputs = self.model( 2025-10-10T01:03:58.7762805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7762884Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7763140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7763230Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7763506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7763600Z return func(*args, **kwargs) 2025-10-10T01:03:58.7763865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7763978Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7764237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7764318Z return func(*args, **kwargs) 2025-10-10T01:03:58.7764582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7764675Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7764688Z 2025-10-10T01:03:58.7764817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7765028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7765106Z res = mod(**inputs) 2025-10-10T01:03:58.7765371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7765452Z outputs = self.model( 2025-10-10T01:03:58.7765714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7765792Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7766035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7766118Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7766399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7766492Z return func(*args, **kwargs) 2025-10-10T01:03:58.7766758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7766871Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7767133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7767216Z return func(*args, **kwargs) 2025-10-10T01:03:58.7767485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7767601Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7767604Z 2025-10-10T01:03:58.7767716Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7767932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7768014Z res = mod(**inputs) 2025-10-10T01:03:58.7768284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7768368Z outputs = self.model( 2025-10-10T01:03:58.7768637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7768727Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7768963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7769044Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7769296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7769366Z return func(*args, **kwargs) 2025-10-10T01:03:58.7769623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7769732Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7769986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7770089Z return func(*args, **kwargs) 2025-10-10T01:03:58.7770353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7770496Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7770500Z 2025-10-10T01:03:58.7770612Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7770822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7770899Z res = mod(**inputs) 2025-10-10T01:03:58.7771164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7771274Z outputs = self.model( 2025-10-10T01:03:58.7771540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7771620Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7771867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7771953Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7772218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7772295Z return func(*args, **kwargs) 2025-10-10T01:03:58.7772568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7772672Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7772964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7773048Z return func(*args, **kwargs) 2025-10-10T01:03:58.7773311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7773409Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7773413Z 2025-10-10T01:03:58.7773523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7773731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7773809Z res = mod(**inputs) 2025-10-10T01:03:58.7774071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7774152Z outputs = self.model( 2025-10-10T01:03:58.7774415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7774498Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7774743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7774831Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7775091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7775163Z return func(*args, **kwargs) 2025-10-10T01:03:58.7775419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7775541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7775545Z 2025-10-10T01:03:58.7775650Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7775859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7775928Z res = mod(**inputs) 2025-10-10T01:03:58.7776186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7776258Z outputs = self.model( 2025-10-10T01:03:58.7776520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7776605Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7776827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7776917Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7777162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7777240Z return func(*args, **kwargs) 2025-10-10T01:03:58.7777491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7777638Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7777872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7777949Z return self.act(input) 2025-10-10T01:03:58.7777953Z 2025-10-10T01:03:58.7778070Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7778284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7778355Z res = mod(**inputs) 2025-10-10T01:03:58.7778627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7778703Z outputs = self.model( 2025-10-10T01:03:58.7778977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7779090Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7779328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7779421Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7779681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7779763Z return func(*args, **kwargs) 2025-10-10T01:03:58.7780032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7780127Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7780131Z 2025-10-10T01:03:58.7780244Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7780470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7780549Z res = mod(**inputs) 2025-10-10T01:03:58.7780818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7780905Z outputs = self.model( 2025-10-10T01:03:58.7781176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7781258Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7781512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7781599Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7781875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7781952Z return func(*args, **kwargs) 2025-10-10T01:03:58.7782223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7782343Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7782610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7782695Z return func(*args, **kwargs) 2025-10-10T01:03:58.7782992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7783120Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7783124Z 2025-10-10T01:03:58.7783235Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7783447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7783526Z res = mod(**inputs) 2025-10-10T01:03:58.7783790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7783871Z outputs = self.model( 2025-10-10T01:03:58.7784153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7784235Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7784486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7784574Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7784846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7784922Z return func(*args, **kwargs) 2025-10-10T01:03:58.7785200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7785305Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7785569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7785684Z return func(*args, **kwargs) 2025-10-10T01:03:58.7785950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7786048Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7786054Z 2025-10-10T01:03:58.7786168Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7786470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7786557Z res = mod(**inputs) 2025-10-10T01:03:58.7786830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7786915Z outputs = self.model( 2025-10-10T01:03:58.7787196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7787275Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7787526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7787612Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7787881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7787958Z return func(*args, **kwargs) 2025-10-10T01:03:58.7788233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7788337Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7788598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7788680Z return func(*args, **kwargs) 2025-10-10T01:03:58.7788945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7789076Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7789080Z 2025-10-10T01:03:58.7789194Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7789411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7789514Z res = mod(**inputs) 2025-10-10T01:03:58.7789775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7789854Z outputs = self.model( 2025-10-10T01:03:58.7790115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7790203Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7790437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7790526Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7790805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7790881Z return func(*args, **kwargs) 2025-10-10T01:03:58.7791150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7791256Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7791513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7791594Z return func(*args, **kwargs) 2025-10-10T01:03:58.7791853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7792004Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7792007Z 2025-10-10T01:03:58.7792151Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7792372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7792442Z res = mod(**inputs) 2025-10-10T01:03:58.7792704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7792787Z outputs = self.model( 2025-10-10T01:03:58.7793049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7793137Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7793372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7793458Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7793724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7793803Z return func(*args, **kwargs) 2025-10-10T01:03:58.7794076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7794180Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7794438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7794520Z return func(*args, **kwargs) 2025-10-10T01:03:58.7794783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7794884Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7794888Z 2025-10-10T01:03:58.7794998Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7795215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7795286Z res = mod(**inputs) 2025-10-10T01:03:58.7795550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7795632Z outputs = self.model( 2025-10-10T01:03:58.7795895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7796002Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7796238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7796321Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7796585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7796659Z return func(*args, **kwargs) 2025-10-10T01:03:58.7796930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7797039Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7797312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7797395Z return func(*args, **kwargs) 2025-10-10T01:03:58.7797659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7797772Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7797775Z 2025-10-10T01:03:58.7797885Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7798104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7798173Z res = mod(**inputs) 2025-10-10T01:03:58.7798438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7798540Z outputs = self.model( 2025-10-10T01:03:58.7798824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7798915Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7799151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7799239Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7799503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7799576Z return func(*args, **kwargs) 2025-10-10T01:03:58.7799847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7799950Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7800206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7800293Z return func(*args, **kwargs) 2025-10-10T01:03:58.7800552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7800697Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7800704Z 2025-10-10T01:03:58.7800813Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7801029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7801099Z res = mod(**inputs) 2025-10-10T01:03:58.7801363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7801443Z outputs = self.model( 2025-10-10T01:03:58.7801705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7801795Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7802034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7802119Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7802406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7802481Z return func(*args, **kwargs) 2025-10-10T01:03:58.7802752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7802856Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7803120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7803193Z return func(*args, **kwargs) 2025-10-10T01:03:58.7803465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7803581Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7803585Z 2025-10-10T01:03:58.7803698Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7803922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7803997Z res = mod(**inputs) 2025-10-10T01:03:58.7804265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7804349Z outputs = self.model( 2025-10-10T01:03:58.7804619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7804707Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7804948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7805071Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7805336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7805411Z return func(*args, **kwargs) 2025-10-10T01:03:58.7805677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7805803Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7805807Z 2025-10-10T01:03:58.7805924Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7806133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7806205Z res = mod(**inputs) 2025-10-10T01:03:58.7806475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7806549Z outputs = self.model( 2025-10-10T01:03:58.7806826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7806907Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7807144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7807249Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7807492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7807571Z return func(*args, **kwargs) 2025-10-10T01:03:58.7807818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7807952Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7808180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7808259Z return self.act(input) 2025-10-10T01:03:58.7808265Z 2025-10-10T01:03:58.7808384Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7808594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7808702Z res = mod(**inputs) 2025-10-10T01:03:58.7808970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7809044Z outputs = self.model( 2025-10-10T01:03:58.7809317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7809398Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7809641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7809730Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7810007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7810092Z return func(*args, **kwargs) 2025-10-10T01:03:58.7810351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7810448Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7810452Z 2025-10-10T01:03:58.7810561Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7810777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7810846Z res = mod(**inputs) 2025-10-10T01:03:58.7811110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7811190Z outputs = self.model( 2025-10-10T01:03:58.7811470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7811575Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7811815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7811903Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7812170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7812244Z return func(*args, **kwargs) 2025-10-10T01:03:58.7812518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7812606Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7812610Z 2025-10-10T01:03:58.7812731Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7812943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7813017Z res = mod(**inputs) 2025-10-10T01:03:58.7813295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7813370Z outputs = self.model( 2025-10-10T01:03:58.7813645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7813725Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7813965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7814057Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7814321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7814403Z return func(*args, **kwargs) 2025-10-10T01:03:58.7814673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7814784Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7815054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7815147Z return func(*args, **kwargs) 2025-10-10T01:03:58.7815426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7815549Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7815553Z 2025-10-10T01:03:58.7815671Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7815884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7815953Z res = mod(**inputs) 2025-10-10T01:03:58.7816232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7816309Z outputs = self.model( 2025-10-10T01:03:58.7816601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7816681Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7816919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7817014Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7817281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7817361Z return func(*args, **kwargs) 2025-10-10T01:03:58.7817634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7817740Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7818033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7818502Z return func(*args, **kwargs) 2025-10-10T01:03:58.7818785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7818875Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7818878Z 2025-10-10T01:03:58.7818998Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7819210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7819281Z res = mod(**inputs) 2025-10-10T01:03:58.7819554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7819630Z outputs = self.model( 2025-10-10T01:03:58.7819914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7819998Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7820239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7820333Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7820594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7820677Z return func(*args, **kwargs) 2025-10-10T01:03:58.7820952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7821058Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7821336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7821409Z return func(*args, **kwargs) 2025-10-10T01:03:58.7821688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7821812Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7821815Z 2025-10-10T01:03:58.7821935Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7822164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7822234Z res = mod(**inputs) 2025-10-10T01:03:58.7822519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7822592Z outputs = self.model( 2025-10-10T01:03:58.7822871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7822951Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7823190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7823304Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7823576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7823657Z return func(*args, **kwargs) 2025-10-10T01:03:58.7823939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7824053Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7824318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7824392Z return func(*args, **kwargs) 2025-10-10T01:03:58.7824667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7824812Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7824860Z 2025-10-10T01:03:58.7824985Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7825197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7825268Z res = mod(**inputs) 2025-10-10T01:03:58.7825555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7825630Z outputs = self.model( 2025-10-10T01:03:58.7825905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7825986Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7826296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7826401Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7826674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7826764Z return func(*args, **kwargs) 2025-10-10T01:03:58.7827030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7827146Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7827405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7827480Z return func(*args, **kwargs) 2025-10-10T01:03:58.7827761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7827858Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7827862Z 2025-10-10T01:03:58.7827987Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7828209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7828285Z res = mod(**inputs) 2025-10-10T01:03:58.7828568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7828645Z outputs = self.model( 2025-10-10T01:03:58.7828957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7829040Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7829287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7829376Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7829635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7829722Z return func(*args, **kwargs) 2025-10-10T01:03:58.7829989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7830121Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7830380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7830457Z return func(*args, **kwargs) 2025-10-10T01:03:58.7830727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7830830Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7830834Z 2025-10-10T01:03:58.7830952Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7831162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7831233Z res = mod(**inputs) 2025-10-10T01:03:58.7831636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7831744Z outputs = self.model( 2025-10-10T01:03:58.7832020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7832100Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7832348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7832435Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7832698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7832781Z return func(*args, **kwargs) 2025-10-10T01:03:58.7833045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7833159Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7833421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7833498Z return func(*args, **kwargs) 2025-10-10T01:03:58.7833775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7833918Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7833922Z 2025-10-10T01:03:58.7834042Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7834255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7834337Z res = mod(**inputs) 2025-10-10T01:03:58.7834601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7834676Z outputs = self.model( 2025-10-10T01:03:58.7834946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7835029Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7835272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7835383Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7835638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7835719Z return func(*args, **kwargs) 2025-10-10T01:03:58.7835980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7836090Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7836344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7836418Z return func(*args, **kwargs) 2025-10-10T01:03:58.7836716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7836807Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7836812Z 2025-10-10T01:03:58.7836930Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7837139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7837217Z res = mod(**inputs) 2025-10-10T01:03:58.7837481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7837555Z outputs = self.model( 2025-10-10T01:03:58.7837824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7837903Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7838170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7838277Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7838535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7838622Z return func(*args, **kwargs) 2025-10-10T01:03:58.7838889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7839022Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7839026Z 2025-10-10T01:03:58.7839136Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7839355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7839427Z res = mod(**inputs) 2025-10-10T01:03:58.7839695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7839781Z outputs = self.model( 2025-10-10T01:03:58.7840050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7840138Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7840382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7840468Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7840744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7840819Z return func(*args, **kwargs) 2025-10-10T01:03:58.7841094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7841222Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7841457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7841543Z return self.act(input) 2025-10-10T01:03:58.7841547Z 2025-10-10T01:03:58.7841659Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7841884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7841969Z res = mod(**inputs) 2025-10-10T01:03:58.7842224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7842300Z outputs = self.model( 2025-10-10T01:03:58.7842554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7842642Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7842880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7842976Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7843255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7843332Z return func(*args, **kwargs) 2025-10-10T01:03:58.7843619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7843709Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7843713Z 2025-10-10T01:03:58.7843831Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7844050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7844119Z res = mod(**inputs) 2025-10-10T01:03:58.7844401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7844470Z outputs = self.model( 2025-10-10T01:03:58.7844767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7844844Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7845073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7845156Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7845403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7845482Z return func(*args, **kwargs) 2025-10-10T01:03:58.7845732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7845841Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7846085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7846160Z return func(*args, **kwargs) 2025-10-10T01:03:58.7846442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7846562Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7846567Z 2025-10-10T01:03:58.7846685Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7846904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7846982Z res = mod(**inputs) 2025-10-10T01:03:58.7847256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7847329Z outputs = self.model( 2025-10-10T01:03:58.7847598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7847675Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7847924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7848011Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7848278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7848380Z return func(*args, **kwargs) 2025-10-10T01:03:58.7848656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7848771Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7849037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7849115Z return func(*args, **kwargs) 2025-10-10T01:03:58.7849406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7849497Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7849516Z 2025-10-10T01:03:58.7849635Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7849848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7849926Z res = mod(**inputs) 2025-10-10T01:03:58.7850191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7850263Z outputs = self.model( 2025-10-10T01:03:58.7850533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7850611Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7850857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7850944Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7851246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7851331Z return func(*args, **kwargs) 2025-10-10T01:03:58.7851600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7851715Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7851977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7852051Z return func(*args, **kwargs) 2025-10-10T01:03:58.7852325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7852447Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7852451Z 2025-10-10T01:03:58.7852569Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7852786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7852864Z res = mod(**inputs) 2025-10-10T01:03:58.7853130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7853205Z outputs = self.model( 2025-10-10T01:03:58.7853482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7853561Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7853807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7853892Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7854156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7854240Z return func(*args, **kwargs) 2025-10-10T01:03:58.7854511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7854624Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7854887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7854989Z return func(*args, **kwargs) 2025-10-10T01:03:58.7855254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7855399Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7855402Z 2025-10-10T01:03:58.7855520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7855731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7855811Z res = mod(**inputs) 2025-10-10T01:03:58.7856094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7856171Z outputs = self.model( 2025-10-10T01:03:58.7856443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7856525Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7856774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7856859Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7857118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7857202Z return func(*args, **kwargs) 2025-10-10T01:03:58.7857470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7857617Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7857877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7857959Z return func(*args, **kwargs) 2025-10-10T01:03:58.7858223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7858315Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7858319Z 2025-10-10T01:03:58.7858439Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7858650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7858726Z res = mod(**inputs) 2025-10-10T01:03:58.7858990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7859062Z outputs = self.model( 2025-10-10T01:03:58.7859334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7859414Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7859657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7859743Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7860007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7860082Z return func(*args, **kwargs) 2025-10-10T01:03:58.7860345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7860456Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7860710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7860795Z return func(*args, **kwargs) 2025-10-10T01:03:58.7861060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7861164Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7861185Z 2025-10-10T01:03:58.7861304Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7861520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7861601Z res = mod(**inputs) 2025-10-10T01:03:58.7861872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7861948Z outputs = self.model( 2025-10-10T01:03:58.7862226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7862307Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7862593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7862682Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7862953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7863032Z return func(*args, **kwargs) 2025-10-10T01:03:58.7863303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7863419Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7863685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7863770Z return func(*args, **kwargs) 2025-10-10T01:03:58.7864058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7864218Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7864230Z 2025-10-10T01:03:58.7864346Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7864564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7864644Z res = mod(**inputs) 2025-10-10T01:03:58.7864919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7865001Z outputs = self.model( 2025-10-10T01:03:58.7865273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7865352Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7865606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7865696Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7865972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7866051Z return func(*args, **kwargs) 2025-10-10T01:03:58.7866420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7866544Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7866812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7866898Z return func(*args, **kwargs) 2025-10-10T01:03:58.7867173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7867265Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7867279Z 2025-10-10T01:03:58.7867396Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7867623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7867702Z res = mod(**inputs) 2025-10-10T01:03:58.7867968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7868075Z outputs = self.model( 2025-10-10T01:03:58.7868338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7868418Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7868666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7868754Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7869020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7869099Z return func(*args, **kwargs) 2025-10-10T01:03:58.7869383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7869521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7869527Z 2025-10-10T01:03:58.7869638Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7869856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7869927Z res = mod(**inputs) 2025-10-10T01:03:58.7870198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7870272Z outputs = self.model( 2025-10-10T01:03:58.7870537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7870626Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7870900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7870996Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7871255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7871331Z return func(*args, **kwargs) 2025-10-10T01:03:58.7871604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7871729Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7871969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7872047Z return self.act(input) 2025-10-10T01:03:58.7872051Z 2025-10-10T01:03:58.7872161Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7872382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7872456Z res = mod(**inputs) 2025-10-10T01:03:58.7872729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7872803Z outputs = self.model( 2025-10-10T01:03:58.7873076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7873155Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7873394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7873488Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7873748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7873830Z return func(*args, **kwargs) 2025-10-10T01:03:58.7874099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7874187Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7874191Z 2025-10-10T01:03:58.7874309Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7874536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7874614Z res = mod(**inputs) 2025-10-10T01:03:58.7874878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7874950Z outputs = self.model( 2025-10-10T01:03:58.7875218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7875297Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7875543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7875657Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7875925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7876002Z return func(*args, **kwargs) 2025-10-10T01:03:58.7876264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.7876359Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.7876362Z 2025-10-10T01:03:58.7876473Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7876684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7876754Z res = mod(**inputs) 2025-10-10T01:03:58.7877017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7877133Z outputs = self.model( 2025-10-10T01:03:58.7877400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7877488Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7877726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7877813Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7878077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7878153Z return func(*args, **kwargs) 2025-10-10T01:03:58.7878426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7878533Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7878807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7878885Z return func(*args, **kwargs) 2025-10-10T01:03:58.7879170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7879300Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7879304Z 2025-10-10T01:03:58.7879414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7879637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7879705Z res = mod(**inputs) 2025-10-10T01:03:58.7879976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7880056Z outputs = self.model( 2025-10-10T01:03:58.7880328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7880417Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7880656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7880748Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7881038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7881113Z return func(*args, **kwargs) 2025-10-10T01:03:58.7881382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7881489Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7881760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7881833Z return func(*args, **kwargs) 2025-10-10T01:03:58.7882107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7882221Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7882225Z 2025-10-10T01:03:58.7882336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7882556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7882624Z res = mod(**inputs) 2025-10-10T01:03:58.7882891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7882972Z outputs = self.model( 2025-10-10T01:03:58.7883243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7883330Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7883565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7883697Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7883946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7884016Z return func(*args, **kwargs) 2025-10-10T01:03:58.7884273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7884739Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7885156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7885573Z return func(*args, **kwargs) 2025-10-10T01:03:58.7886301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7886775Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7886956Z 2025-10-10T01:03:58.7887083Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7887472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7887836Z res = mod(**inputs) 2025-10-10T01:03:58.7888245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7888657Z outputs = self.model( 2025-10-10T01:03:58.7889048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7889466Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7889849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7890245Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7890660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7891068Z return func(*args, **kwargs) 2025-10-10T01:03:58.7891466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7891909Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7892344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7892727Z return func(*args, **kwargs) 2025-10-10T01:03:58.7893099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7893554Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7893758Z 2025-10-10T01:03:58.7893865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7894233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7894569Z res = mod(**inputs) 2025-10-10T01:03:58.7894943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7895328Z outputs = self.model( 2025-10-10T01:03:58.7895699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7896103Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7896488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7896863Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7897248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7897630Z return func(*args, **kwargs) 2025-10-10T01:03:58.7898047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7898499Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7898932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7899340Z return func(*args, **kwargs) 2025-10-10T01:03:58.7899738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7900170Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7900326Z 2025-10-10T01:03:58.7900443Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7900835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7901183Z res = mod(**inputs) 2025-10-10T01:03:58.7901571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7901980Z outputs = self.model( 2025-10-10T01:03:58.7902376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7902796Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7903183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7903586Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7903995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7904402Z return func(*args, **kwargs) 2025-10-10T01:03:58.7904802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7905249Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7905685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7906083Z return func(*args, **kwargs) 2025-10-10T01:03:58.7906563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7907064Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7907233Z 2025-10-10T01:03:58.7907365Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7907760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7908124Z res = mod(**inputs) 2025-10-10T01:03:58.7908510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7908921Z outputs = self.model( 2025-10-10T01:03:58.7909310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7909719Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7910120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7910520Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7910945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7911365Z return func(*args, **kwargs) 2025-10-10T01:03:58.7911766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7912212Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7912654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7913070Z return func(*args, **kwargs) 2025-10-10T01:03:58.7913483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7913982Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7914188Z 2025-10-10T01:03:58.7914300Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7914692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7915042Z res = mod(**inputs) 2025-10-10T01:03:58.7915423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7915840Z outputs = self.model( 2025-10-10T01:03:58.7916230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7916654Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7917036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7917433Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7917844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7918257Z return func(*args, **kwargs) 2025-10-10T01:03:58.7918660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7919093Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7919525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7919935Z return func(*args, **kwargs) 2025-10-10T01:03:58.7920355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.7920832Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.7920977Z 2025-10-10T01:03:58.7921084Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7921464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7921816Z res = mod(**inputs) 2025-10-10T01:03:58.7922238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7922657Z outputs = self.model( 2025-10-10T01:03:58.7923047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7923467Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7923847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7924246Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7924649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7925051Z return func(*args, **kwargs) 2025-10-10T01:03:58.7925434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7925903Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7926090Z 2025-10-10T01:03:58.7926211Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7926592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7926940Z res = mod(**inputs) 2025-10-10T01:03:58.7927336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7927755Z outputs = self.model( 2025-10-10T01:03:58.7928150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7928611Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7928996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7929371Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7929762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7930137Z return func(*args, **kwargs) 2025-10-10T01:03:58.7930517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.7930985Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.7931410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.7931957Z return self.act(input) 2025-10-10T01:03:58.7932076Z 2025-10-10T01:03:58.7932185Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7932568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7932900Z res = mod(**inputs) 2025-10-10T01:03:58.7933281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7933694Z outputs = self.model( 2025-10-10T01:03:58.7934063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7934455Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7934818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7935194Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7935577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7935993Z return func(*args, **kwargs) 2025-10-10T01:03:58.7936402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.7936804Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.7936998Z 2025-10-10T01:03:58.7937110Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7937472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7937803Z res = mod(**inputs) 2025-10-10T01:03:58.7938173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7938566Z outputs = self.model( 2025-10-10T01:03:58.7938925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7939321Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7939713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7940089Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7940480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7940858Z return func(*args, **kwargs) 2025-10-10T01:03:58.7941236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7941654Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7942063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7942436Z return func(*args, **kwargs) 2025-10-10T01:03:58.7942834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7943347Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7943530Z 2025-10-10T01:03:58.7943652Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7944039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7944382Z res = mod(**inputs) 2025-10-10T01:03:58.7944769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7945179Z outputs = self.model( 2025-10-10T01:03:58.7945571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7945985Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7946431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7946860Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7947297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7947749Z return func(*args, **kwargs) 2025-10-10T01:03:58.7948145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7948593Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7949032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7949436Z return func(*args, **kwargs) 2025-10-10T01:03:58.7949838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.7950257Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.7950415Z 2025-10-10T01:03:58.7950529Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7950921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7951273Z res = mod(**inputs) 2025-10-10T01:03:58.7951659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7952079Z outputs = self.model( 2025-10-10T01:03:58.7952461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7952877Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7953252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7953635Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7954045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7954450Z return func(*args, **kwargs) 2025-10-10T01:03:58.7954870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7955316Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7955748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7956124Z return func(*args, **kwargs) 2025-10-10T01:03:58.7956489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.7956908Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.7957075Z 2025-10-10T01:03:58.7957187Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7957535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7957856Z res = mod(**inputs) 2025-10-10T01:03:58.7958228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7958629Z outputs = self.model( 2025-10-10T01:03:58.7958982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7959366Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7959718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7960091Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7960471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7960840Z return func(*args, **kwargs) 2025-10-10T01:03:58.7961216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7961629Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7962032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7962399Z return func(*args, **kwargs) 2025-10-10T01:03:58.7962771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.7963222Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.7963417Z 2025-10-10T01:03:58.7963533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7963903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7964226Z res = mod(**inputs) 2025-10-10T01:03:58.7964596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7965033Z outputs = self.model( 2025-10-10T01:03:58.7965400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7965790Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7966146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7966543Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7966935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7967321Z return func(*args, **kwargs) 2025-10-10T01:03:58.7967723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7968173Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7968612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7969001Z return func(*args, **kwargs) 2025-10-10T01:03:58.7969403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.7969793Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.7969947Z 2025-10-10T01:03:58.7970054Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7970417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7970742Z res = mod(**inputs) 2025-10-10T01:03:58.7971101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7971478Z outputs = self.model( 2025-10-10T01:03:58.7971849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7972228Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7972623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7972984Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7973371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7973751Z return func(*args, **kwargs) 2025-10-10T01:03:58.7974127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7974541Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7974945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7975332Z return func(*args, **kwargs) 2025-10-10T01:03:58.7975734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.7976185Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.7976352Z 2025-10-10T01:03:58.7976472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7976839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7977168Z res = mod(**inputs) 2025-10-10T01:03:58.7977527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7977911Z outputs = self.model( 2025-10-10T01:03:58.7978267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7978669Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7979028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7979420Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7979835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7980229Z return func(*args, **kwargs) 2025-10-10T01:03:58.7980625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.7981080Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.7981506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7981906Z return func(*args, **kwargs) 2025-10-10T01:03:58.7982304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.7982778Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.7982972Z 2025-10-10T01:03:58.7983093Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.7983505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.7983852Z res = mod(**inputs) 2025-10-10T01:03:58.7984261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.7984681Z outputs = self.model( 2025-10-10T01:03:58.7985076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.7985504Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.7985885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.7986377Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.7986828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.7987343Z return func(*args, **kwargs) 2025-10-10T01:03:58.7987750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8000328Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8000982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8001419Z return func(*args, **kwargs) 2025-10-10T01:03:58.8001842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8002280Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8002438Z 2025-10-10T01:03:58.8002566Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8002968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8003333Z res = mod(**inputs) 2025-10-10T01:03:58.8003763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8004209Z outputs = self.model( 2025-10-10T01:03:58.8004629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8005079Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8005468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8005886Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8006317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8006741Z return func(*args, **kwargs) 2025-10-10T01:03:58.8007160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8007659Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8007870Z 2025-10-10T01:03:58.8007996Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8008418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8008887Z res = mod(**inputs) 2025-10-10T01:03:58.8009280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8009714Z outputs = self.model( 2025-10-10T01:03:58.8010122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8010553Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8010944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8011358Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8011831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8012256Z return func(*args, **kwargs) 2025-10-10T01:03:58.8012678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8013155Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8013600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8013988Z return self.act(input) 2025-10-10T01:03:58.8014115Z 2025-10-10T01:03:58.8014245Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8014657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8015015Z res = mod(**inputs) 2025-10-10T01:03:58.8015446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8015897Z outputs = self.model( 2025-10-10T01:03:58.8016297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8016728Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8017130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8017550Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8017984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8018427Z return func(*args, **kwargs) 2025-10-10T01:03:58.8018844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8019296Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8019463Z 2025-10-10T01:03:58.8019586Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8020001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8020369Z res = mod(**inputs) 2025-10-10T01:03:58.8020770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8021201Z outputs = self.model( 2025-10-10T01:03:58.8021608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8022050Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8022440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8022854Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8023287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8023716Z return func(*args, **kwargs) 2025-10-10T01:03:58.8024134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.8024576Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.8024759Z 2025-10-10T01:03:58.8024879Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8025283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8025641Z res = mod(**inputs) 2025-10-10T01:03:58.8026041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8026587Z outputs = self.model( 2025-10-10T01:03:58.8027008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8027485Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8027908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8028317Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8028746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8029174Z return func(*args, **kwargs) 2025-10-10T01:03:58.8029595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8030054Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8030496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8030926Z return func(*args, **kwargs) 2025-10-10T01:03:58.8031357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8032038Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8032230Z 2025-10-10T01:03:58.8032350Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8032755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8033119Z res = mod(**inputs) 2025-10-10T01:03:58.8033519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8033960Z outputs = self.model( 2025-10-10T01:03:58.8034352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8034788Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8035176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8035593Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8036019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8036434Z return func(*args, **kwargs) 2025-10-10T01:03:58.8036853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8037324Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8037773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8038181Z return func(*args, **kwargs) 2025-10-10T01:03:58.8038604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.8038998Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.8039135Z 2025-10-10T01:03:58.8039248Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8039610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8039943Z res = mod(**inputs) 2025-10-10T01:03:58.8040330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8040792Z outputs = self.model( 2025-10-10T01:03:58.8041187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8041606Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8041988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8042367Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8042756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8043143Z return func(*args, **kwargs) 2025-10-10T01:03:58.8043541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8043966Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8044382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8044767Z return func(*args, **kwargs) 2025-10-10T01:03:58.8045148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8045588Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8045765Z 2025-10-10T01:03:58.8045873Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8046235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8046606Z res = mod(**inputs) 2025-10-10T01:03:58.8046974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8047361Z outputs = self.model( 2025-10-10T01:03:58.8047732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8048139Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8048491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8048853Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8049240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8049615Z return func(*args, **kwargs) 2025-10-10T01:03:58.8049987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8050400Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8050793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8051169Z return func(*args, **kwargs) 2025-10-10T01:03:58.8051538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.8051983Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.8052171Z 2025-10-10T01:03:58.8052282Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8052638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8052960Z res = mod(**inputs) 2025-10-10T01:03:58.8053314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8053700Z outputs = self.model( 2025-10-10T01:03:58.8054052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8054440Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8054809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8055176Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8055557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8055922Z return func(*args, **kwargs) 2025-10-10T01:03:58.8056289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8056698Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8057103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8057479Z return func(*args, **kwargs) 2025-10-10T01:03:58.8057841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.8058227Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.8058366Z 2025-10-10T01:03:58.8058480Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8058828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8059131Z res = mod(**inputs) 2025-10-10T01:03:58.8059474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8059841Z outputs = self.model( 2025-10-10T01:03:58.8060186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8060595Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8060936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8061296Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8061669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8062045Z return func(*args, **kwargs) 2025-10-10T01:03:58.8062405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8062812Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8063210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8063584Z return func(*args, **kwargs) 2025-10-10T01:03:58.8063955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.8064360Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.8064521Z 2025-10-10T01:03:58.8064626Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8064992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8065317Z res = mod(**inputs) 2025-10-10T01:03:58.8065672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8066064Z outputs = self.model( 2025-10-10T01:03:58.8066504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8066911Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8067276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8067653Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8068063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8068437Z return func(*args, **kwargs) 2025-10-10T01:03:58.8068831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8069238Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8069635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8069998Z return func(*args, **kwargs) 2025-10-10T01:03:58.8070353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.8070778Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.8070953Z 2025-10-10T01:03:58.8071062Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8071424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8071739Z res = mod(**inputs) 2025-10-10T01:03:58.8072084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8072443Z outputs = self.model( 2025-10-10T01:03:58.8072798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8073168Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8073499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8073854Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8074237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8074613Z return func(*args, **kwargs) 2025-10-10T01:03:58.8074970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8075358Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8075749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8076107Z return func(*args, **kwargs) 2025-10-10T01:03:58.8076466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8076847Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8076982Z 2025-10-10T01:03:58.8077084Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8077437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8077759Z res = mod(**inputs) 2025-10-10T01:03:58.8078114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8078485Z outputs = self.model( 2025-10-10T01:03:58.8078844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8079226Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8079566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8079917Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8080291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8080672Z return func(*args, **kwargs) 2025-10-10T01:03:58.8081050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8081491Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8081668Z 2025-10-10T01:03:58.8081782Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8082139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8082485Z res = mod(**inputs) 2025-10-10T01:03:58.8082825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8083188Z outputs = self.model( 2025-10-10T01:03:58.8083525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8083895Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8084234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8084590Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8084977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8085330Z return func(*args, **kwargs) 2025-10-10T01:03:58.8085688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8086103Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8086480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8086813Z return self.act(input) 2025-10-10T01:03:58.8086922Z 2025-10-10T01:03:58.8087022Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8087371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8087681Z res = mod(**inputs) 2025-10-10T01:03:58.8088058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8088416Z outputs = self.model( 2025-10-10T01:03:58.8088763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8089136Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8089481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8089861Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8090258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8090617Z return func(*args, **kwargs) 2025-10-10T01:03:58.8090974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8091356Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8091489Z 2025-10-10T01:03:58.8091593Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8091940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8092251Z res = mod(**inputs) 2025-10-10T01:03:58.8092491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8092565Z outputs = self.model( 2025-10-10T01:03:58.8092800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8092880Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8093093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8093172Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8093411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8093482Z return func(*args, **kwargs) 2025-10-10T01:03:58.8093728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8093844Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8094083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8094151Z return func(*args, **kwargs) 2025-10-10T01:03:58.8094384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8094502Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8094505Z 2025-10-10T01:03:58.8094604Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8094801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8094890Z res = mod(**inputs) 2025-10-10T01:03:58.8095130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8095204Z outputs = self.model( 2025-10-10T01:03:58.8095441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8095521Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8095731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8095814Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8096044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8096111Z return func(*args, **kwargs) 2025-10-10T01:03:58.8096385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8096482Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8096720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8096787Z return func(*args, **kwargs) 2025-10-10T01:03:58.8097023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.8097110Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.8097113Z 2025-10-10T01:03:58.8097212Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8097407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8097468Z res = mod(**inputs) 2025-10-10T01:03:58.8097709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8097784Z outputs = self.model( 2025-10-10T01:03:58.8098022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8098103Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8098315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8098397Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8098627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8098694Z return func(*args, **kwargs) 2025-10-10T01:03:58.8098938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8099033Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8099275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8099341Z return func(*args, **kwargs) 2025-10-10T01:03:58.8099577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8099710Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8099714Z 2025-10-10T01:03:58.8099815Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8100012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8100076Z res = mod(**inputs) 2025-10-10T01:03:58.8100321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8100389Z outputs = self.model( 2025-10-10T01:03:58.8100625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8100724Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8100936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8101021Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8101251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8101318Z return func(*args, **kwargs) 2025-10-10T01:03:58.8101563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8101661Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8101902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8101970Z return func(*args, **kwargs) 2025-10-10T01:03:58.8102246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.8102393Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.8102398Z 2025-10-10T01:03:58.8102500Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8102700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8102764Z res = mod(**inputs) 2025-10-10T01:03:58.8103015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8103083Z outputs = self.model( 2025-10-10T01:03:58.8103325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8103405Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8103626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8103713Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8103950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8104020Z return func(*args, **kwargs) 2025-10-10T01:03:58.8104269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8104366Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8104609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8104676Z return func(*args, **kwargs) 2025-10-10T01:03:58.8104924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.8105014Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.8105018Z 2025-10-10T01:03:58.8105122Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8105324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8105407Z res = mod(**inputs) 2025-10-10T01:03:58.8105656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8105724Z outputs = self.model( 2025-10-10T01:03:58.8105964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8106044Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8106342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8106433Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8106696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8106767Z return func(*args, **kwargs) 2025-10-10T01:03:58.8107046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8107157Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8107439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8107514Z return func(*args, **kwargs) 2025-10-10T01:03:58.8107788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.8107894Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.8107898Z 2025-10-10T01:03:58.8108009Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8108267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8108341Z res = mod(**inputs) 2025-10-10T01:03:58.8108615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8108690Z outputs = self.model( 2025-10-10T01:03:58.8108966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8109049Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8109275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8109363Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8109629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8109704Z return func(*args, **kwargs) 2025-10-10T01:03:58.8109999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8110105Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8110378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8110455Z return func(*args, **kwargs) 2025-10-10T01:03:58.8110739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.8110881Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.8110884Z 2025-10-10T01:03:58.8110996Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8111225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8111295Z res = mod(**inputs) 2025-10-10T01:03:58.8111586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8111660Z outputs = self.model( 2025-10-10T01:03:58.8111932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8112033Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8112257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8112346Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8112591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8112669Z return func(*args, **kwargs) 2025-10-10T01:03:58.8112919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8113020Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8113293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8113367Z return func(*args, **kwargs) 2025-10-10T01:03:58.8113630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8113718Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8113723Z 2025-10-10T01:03:58.8113828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8114039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8114104Z res = mod(**inputs) 2025-10-10T01:03:58.8114368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8114437Z outputs = self.model( 2025-10-10T01:03:58.8114722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8114806Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8115029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8115118Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8115378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8115461Z return func(*args, **kwargs) 2025-10-10T01:03:58.8115733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8115863Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8115867Z 2025-10-10T01:03:58.8115986Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8116211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8116292Z res = mod(**inputs) 2025-10-10T01:03:58.8116575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8116649Z outputs = self.model( 2025-10-10T01:03:58.8116930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8117006Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8117257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8117339Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8117591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8117663Z return func(*args, **kwargs) 2025-10-10T01:03:58.8117917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8118045Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8118261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8118369Z return self.act(input) 2025-10-10T01:03:58.8118373Z 2025-10-10T01:03:58.8118482Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8118693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8118768Z res = mod(**inputs) 2025-10-10T01:03:58.8119029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8119109Z outputs = self.model( 2025-10-10T01:03:58.8119372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8119467Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8119710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8119798Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8120063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8120138Z return func(*args, **kwargs) 2025-10-10T01:03:58.8120405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8120494Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8120498Z 2025-10-10T01:03:58.8120608Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8120827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8120930Z res = mod(**inputs) 2025-10-10T01:03:58.8121209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8121283Z outputs = self.model( 2025-10-10T01:03:58.8121549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8121639Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8121880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8121974Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8122239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8122315Z return func(*args, **kwargs) 2025-10-10T01:03:58.8122590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.8122681Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.8122685Z 2025-10-10T01:03:58.8122801Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8123014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8123093Z res = mod(**inputs) 2025-10-10T01:03:58.8123363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8123436Z outputs = self.model( 2025-10-10T01:03:58.8123709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8123787Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8124034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8124119Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8124389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8124471Z return func(*args, **kwargs) 2025-10-10T01:03:58.8124734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8124864Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8125121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8125201Z return func(*args, **kwargs) 2025-10-10T01:03:58.8125465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8125585Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8125589Z 2025-10-10T01:03:58.8125705Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8125939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8126016Z res = mod(**inputs) 2025-10-10T01:03:58.8126282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8126357Z outputs = self.model( 2025-10-10T01:03:58.8126631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8126710Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8126958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8127042Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8127305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8127421Z return func(*args, **kwargs) 2025-10-10T01:03:58.8127685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8127798Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8128060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8128142Z return func(*args, **kwargs) 2025-10-10T01:03:58.8128412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.8128503Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.8128507Z 2025-10-10T01:03:58.8128627Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8128841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8128919Z res = mod(**inputs) 2025-10-10T01:03:58.8129195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8129271Z outputs = self.model( 2025-10-10T01:03:58.8129555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8129637Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8129880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8129965Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8130227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8130302Z return func(*args, **kwargs) 2025-10-10T01:03:58.8130564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8130680Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8130939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8131021Z return func(*args, **kwargs) 2025-10-10T01:03:58.8131305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8131426Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8131429Z 2025-10-10T01:03:58.8131746Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8131963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8132043Z res = mod(**inputs) 2025-10-10T01:03:58.8132307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8132381Z outputs = self.model( 2025-10-10T01:03:58.8132694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8132776Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8133023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8133111Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8133379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8133454Z return func(*args, **kwargs) 2025-10-10T01:03:58.8133722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8133834Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8134094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8134228Z return func(*args, **kwargs) 2025-10-10T01:03:58.8134497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.8134644Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.8134650Z 2025-10-10T01:03:58.8134773Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8134983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8135063Z res = mod(**inputs) 2025-10-10T01:03:58.8135330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8135414Z outputs = self.model( 2025-10-10T01:03:58.8135647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8135723Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8135943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8136020Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8136263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8136335Z return func(*args, **kwargs) 2025-10-10T01:03:58.8136577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8136681Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8136922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8136997Z return func(*args, **kwargs) 2025-10-10T01:03:58.8137237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.8137329Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.8137340Z 2025-10-10T01:03:58.8137441Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8137634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8137734Z res = mod(**inputs) 2025-10-10T01:03:58.8137978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8138053Z outputs = self.model( 2025-10-10T01:03:58.8138296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8138370Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8138604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8138684Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8138936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8139005Z return func(*args, **kwargs) 2025-10-10T01:03:58.8139242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8139348Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8139580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8139653Z return func(*args, **kwargs) 2025-10-10T01:03:58.8139893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.8139989Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.8139999Z 2025-10-10T01:03:58.8140115Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8140332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8140405Z res = mod(**inputs) 2025-10-10T01:03:58.8140648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8140722Z outputs = self.model( 2025-10-10T01:03:58.8140963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8141035Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8141258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8141337Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8141579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8141651Z return func(*args, **kwargs) 2025-10-10T01:03:58.8141894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8142000Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8142243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8142322Z return func(*args, **kwargs) 2025-10-10T01:03:58.8142572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.8142709Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.8142712Z 2025-10-10T01:03:58.8142816Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8143016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8143093Z res = mod(**inputs) 2025-10-10T01:03:58.8143344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8143421Z outputs = self.model( 2025-10-10T01:03:58.8143668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8143763Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8144002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8144089Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8144357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8144433Z return func(*args, **kwargs) 2025-10-10T01:03:58.8144698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8144828Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8145084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8145167Z return func(*args, **kwargs) 2025-10-10T01:03:58.8145434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8145533Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8145537Z 2025-10-10T01:03:58.8145647Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8145859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8145938Z res = mod(**inputs) 2025-10-10T01:03:58.8146249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8146362Z outputs = self.model( 2025-10-10T01:03:58.8146650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8146733Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8146975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8147062Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8147327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8147404Z return func(*args, **kwargs) 2025-10-10T01:03:58.8147676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8147810Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8147820Z 2025-10-10T01:03:58.8147928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8148137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8148204Z res = mod(**inputs) 2025-10-10T01:03:58.8148455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8148526Z outputs = self.model( 2025-10-10T01:03:58.8148768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8148854Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8149072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8149162Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8149405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8149480Z return func(*args, **kwargs) 2025-10-10T01:03:58.8149748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8149867Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8150104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8150175Z return self.act(input) 2025-10-10T01:03:58.8150179Z 2025-10-10T01:03:58.8150288Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8150482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8150547Z res = mod(**inputs) 2025-10-10T01:03:58.8150795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8150863Z outputs = self.model( 2025-10-10T01:03:58.8151130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8151207Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8151429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8151521Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8151762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8151838Z return func(*args, **kwargs) 2025-10-10T01:03:58.8152081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8152163Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8152174Z 2025-10-10T01:03:58.8152284Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8152512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8152612Z res = mod(**inputs) 2025-10-10T01:03:58.8152877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8152958Z outputs = self.model( 2025-10-10T01:03:58.8153222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8153302Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8153544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8153639Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8153888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8153960Z return func(*args, **kwargs) 2025-10-10T01:03:58.8154219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8154331Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8154567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8154644Z return func(*args, **kwargs) 2025-10-10T01:03:58.8154887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8155003Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8155007Z 2025-10-10T01:03:58.8155108Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8155299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8155372Z res = mod(**inputs) 2025-10-10T01:03:58.8155619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8155697Z outputs = self.model( 2025-10-10T01:03:58.8155944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8156038Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8156266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8156347Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8156596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8156667Z return func(*args, **kwargs) 2025-10-10T01:03:58.8156914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8157019Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8157294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8157372Z return func(*args, **kwargs) 2025-10-10T01:03:58.8157616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.8157705Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.8157709Z 2025-10-10T01:03:58.8157809Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8158005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8158078Z res = mod(**inputs) 2025-10-10T01:03:58.8158319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8158393Z outputs = self.model( 2025-10-10T01:03:58.8158651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8158742Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8158965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8159046Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8159289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8159359Z return func(*args, **kwargs) 2025-10-10T01:03:58.8159599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8159704Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8159941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8160018Z return func(*args, **kwargs) 2025-10-10T01:03:58.8160265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8160384Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8160388Z 2025-10-10T01:03:58.8160491Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8160692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8160766Z res = mod(**inputs) 2025-10-10T01:03:58.8161024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8161106Z outputs = self.model( 2025-10-10T01:03:58.8161372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8161451Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8161697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8161786Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8162054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8162156Z return func(*args, **kwargs) 2025-10-10T01:03:58.8162411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8162512Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8162765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8162842Z return func(*args, **kwargs) 2025-10-10T01:03:58.8163084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.8163227Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.8163232Z 2025-10-10T01:03:58.8163351Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8163545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8163620Z res = mod(**inputs) 2025-10-10T01:03:58.8163863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8163938Z outputs = self.model( 2025-10-10T01:03:58.8164179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8164258Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8164475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8164553Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8164821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8164906Z return func(*args, **kwargs) 2025-10-10T01:03:58.8165153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8165253Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8165488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8165563Z return func(*args, **kwargs) 2025-10-10T01:03:58.8165803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.8165897Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.8165900Z 2025-10-10T01:03:58.8166002Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8166196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8166272Z res = mod(**inputs) 2025-10-10T01:03:58.8166516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8166594Z outputs = self.model( 2025-10-10T01:03:58.8166842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8166922Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8167157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8167235Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8167477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8167546Z return func(*args, **kwargs) 2025-10-10T01:03:58.8167800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8167897Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8168132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8168230Z return func(*args, **kwargs) 2025-10-10T01:03:58.8168478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.8168585Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.8168588Z 2025-10-10T01:03:58.8168693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8168899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8168966Z res = mod(**inputs) 2025-10-10T01:03:58.8169215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8169308Z outputs = self.model( 2025-10-10T01:03:58.8169553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8169634Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8169851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8169929Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8170175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8170243Z return func(*args, **kwargs) 2025-10-10T01:03:58.8170494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8170592Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8170872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8170949Z return func(*args, **kwargs) 2025-10-10T01:03:58.8171190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.8171325Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.8171329Z 2025-10-10T01:03:58.8171429Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8171633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8171697Z res = mod(**inputs) 2025-10-10T01:03:58.8171939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8172013Z outputs = self.model( 2025-10-10T01:03:58.8172255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8172338Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8172557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8172638Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8172883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8172952Z return func(*args, **kwargs) 2025-10-10T01:03:58.8173201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8173297Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8173533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8173609Z return func(*args, **kwargs) 2025-10-10T01:03:58.8173849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8173940Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8173944Z 2025-10-10T01:03:58.8174061Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8174262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8174330Z res = mod(**inputs) 2025-10-10T01:03:58.8174574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8174650Z outputs = self.model( 2025-10-10T01:03:58.8174894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8174973Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8175191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8175284Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8175528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8175599Z return func(*args, **kwargs) 2025-10-10T01:03:58.8175848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8175964Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8175968Z 2025-10-10T01:03:58.8176075Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8176271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8176336Z res = mod(**inputs) 2025-10-10T01:03:58.8176601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8176686Z outputs = self.model( 2025-10-10T01:03:58.8176937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8177011Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8177227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8177316Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8177551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8177626Z return func(*args, **kwargs) 2025-10-10T01:03:58.8177869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8177983Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8178206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8178277Z return self.act(input) 2025-10-10T01:03:58.8178281Z 2025-10-10T01:03:58.8178389Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8178586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8178668Z res = mod(**inputs) 2025-10-10T01:03:58.8178905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8178973Z outputs = self.model( 2025-10-10T01:03:58.8179218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8179292Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8179518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8179602Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8179844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8179922Z return func(*args, **kwargs) 2025-10-10T01:03:58.8180187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8180279Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8180283Z 2025-10-10T01:03:58.8180387Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8180592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8180659Z res = mod(**inputs) 2025-10-10T01:03:58.8180905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8180984Z outputs = self.model( 2025-10-10T01:03:58.8181265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8181354Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8181593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8181680Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8181950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8182024Z return func(*args, **kwargs) 2025-10-10T01:03:58.8182294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-10-10T01:03:58.8182384Z hidden_states = residual + hidden_states 2025-10-10T01:03:58.8182388Z 2025-10-10T01:03:58.8182498Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8182770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8182842Z res = mod(**inputs) 2025-10-10T01:03:58.8183113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8183186Z outputs = self.model( 2025-10-10T01:03:58.8183445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8183523Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8183748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8183834Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8184082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8184162Z return func(*args, **kwargs) 2025-10-10T01:03:58.8184431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8184538Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8184806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8184883Z return func(*args, **kwargs) 2025-10-10T01:03:58.8185155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8185278Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8185282Z 2025-10-10T01:03:58.8185392Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8185612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8185682Z res = mod(**inputs) 2025-10-10T01:03:58.8185958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8186033Z outputs = self.model( 2025-10-10T01:03:58.8186389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8186505Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8186750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8186848Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8187114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8187200Z return func(*args, **kwargs) 2025-10-10T01:03:58.8187483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8187591Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8187886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8187964Z return func(*args, **kwargs) 2025-10-10T01:03:58.8188241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-10-10T01:03:58.8188331Z key_states = self.k_proj(current_states) 2025-10-10T01:03:58.8188335Z 2025-10-10T01:03:58.8188453Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8188670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8188742Z res = mod(**inputs) 2025-10-10T01:03:58.8189026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8189096Z outputs = self.model( 2025-10-10T01:03:58.8190264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8190343Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8190565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8190661Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8190920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8191007Z return func(*args, **kwargs) 2025-10-10T01:03:58.8191281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8191387Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8191664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8191744Z return func(*args, **kwargs) 2025-10-10T01:03:58.8192030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-10-10T01:03:58.8192151Z query_states = self.q_proj(hidden_states) * self.scaling 2025-10-10T01:03:58.8192156Z 2025-10-10T01:03:58.8192274Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8192488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8192569Z res = mod(**inputs) 2025-10-10T01:03:58.8192818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8192888Z outputs = self.model( 2025-10-10T01:03:58.8193139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8193215Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8193445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8193539Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8193803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8193905Z return func(*args, **kwargs) 2025-10-10T01:03:58.8194179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8194283Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8194560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8194634Z return func(*args, **kwargs) 2025-10-10T01:03:58.8194905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-10-10T01:03:58.8195071Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-10-10T01:03:58.8195076Z 2025-10-10T01:03:58.8195194Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8195408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8195480Z res = mod(**inputs) 2025-10-10T01:03:58.8195759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8195829Z outputs = self.model( 2025-10-10T01:03:58.8196086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8196162Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8196387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8196476Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8196756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8196837Z return func(*args, **kwargs) 2025-10-10T01:03:58.8197089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8197198Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8197446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8197517Z return func(*args, **kwargs) 2025-10-10T01:03:58.8197777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-10-10T01:03:58.8197866Z value_states = self.v_proj(current_states) 2025-10-10T01:03:58.8197870Z 2025-10-10T01:03:58.8197981Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8198188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8198253Z res = mod(**inputs) 2025-10-10T01:03:58.8198514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8198586Z outputs = self.model( 2025-10-10T01:03:58.8198874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8198954Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8199193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8199286Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8199556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8199641Z return func(*args, **kwargs) 2025-10-10T01:03:58.8199929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8200040Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8200329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8200404Z return func(*args, **kwargs) 2025-10-10T01:03:58.8200682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-10-10T01:03:58.8200780Z attn_output = torch.bmm(attn_probs, value_states) 2025-10-10T01:03:58.8200784Z 2025-10-10T01:03:58.8200891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8201092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8201158Z res = mod(**inputs) 2025-10-10T01:03:58.8201432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8201504Z outputs = self.model( 2025-10-10T01:03:58.8201757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8201833Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8202063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8202145Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8202387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8202466Z return func(*args, **kwargs) 2025-10-10T01:03:58.8202712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8202852Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8203095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8203167Z return func(*args, **kwargs) 2025-10-10T01:03:58.8203423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-10-10T01:03:58.8203554Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-10-10T01:03:58.8203558Z 2025-10-10T01:03:58.8203668Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8203867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8203933Z res = mod(**inputs) 2025-10-10T01:03:58.8204196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8204271Z outputs = self.model( 2025-10-10T01:03:58.8204545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8204623Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8204867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8204956Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8205212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8205295Z return func(*args, **kwargs) 2025-10-10T01:03:58.8205563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-10-10T01:03:58.8205678Z hidden_states, self_attn_weights = self.self_attn( 2025-10-10T01:03:58.8205943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8206021Z return func(*args, **kwargs) 2025-10-10T01:03:58.8206306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-10-10T01:03:58.8206395Z attn_output = self.out_proj(attn_output) 2025-10-10T01:03:58.8206417Z 2025-10-10T01:03:58.8206535Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8206747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8206824Z res = mod(**inputs) 2025-10-10T01:03:58.8207088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8207162Z outputs = self.model( 2025-10-10T01:03:58.8207431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8207510Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8207775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8207862Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8208119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8208203Z return func(*args, **kwargs) 2025-10-10T01:03:58.8208466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8208600Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8208604Z 2025-10-10T01:03:58.8208714Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8208925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8209003Z res = mod(**inputs) 2025-10-10T01:03:58.8209305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8209388Z outputs = self.model( 2025-10-10T01:03:58.8209655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8209743Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8209980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8210065Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8210333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8210410Z return func(*args, **kwargs) 2025-10-10T01:03:58.8210680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-10-10T01:03:58.8210810Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-10-10T01:03:58.8211044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:03:58.8211127Z return self.act(input) 2025-10-10T01:03:58.8211131Z 2025-10-10T01:03:58.8211243Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8211462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8211533Z res = mod(**inputs) 2025-10-10T01:03:58.8211799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-10-10T01:03:58.8211883Z outputs = self.model( 2025-10-10T01:03:58.8212156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-10-10T01:03:58.8212245Z layer_outputs = decoder_layer( 2025-10-10T01:03:58.8212491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:03:58.8212592Z return super().__call__(*args, **kwargs) 2025-10-10T01:03:58.8212861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-10-10T01:03:58.8212971Z return func(*args, **kwargs) 2025-10-10T01:03:58.8213256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-10-10T01:03:58.8213346Z hidden_states = self.fc2(hidden_states) 2025-10-10T01:03:58.8213350Z 2025-10-10T01:03:58.8213466Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8213680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8213751Z res = mod(**inputs) 2025-10-10T01:03:58.8214033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-10-10T01:03:58.8214124Z logits = self.lm_head(outputs[0]) 2025-10-10T01:03:58.8214146Z 2025-10-10T01:03:58.8214269Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:03:58.8214484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:03:58.8214565Z res = mod(**inputs) 2025-10-10T01:03:58.8214839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 689, in forward 2025-10-10T01:03:58.8214920Z loss = self.loss_function( 2025-10-10T01:03:58.8215204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-10-10T01:03:58.8215391Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-10-10T01:03:58.8215672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-10-10T01:03:58.8215916Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-10-10T01:03:58.8215920Z 2025-10-10T01:04:11.7498791Z Compilation time (from dynamo_timed): 26.658674411 2025-10-10T01:04:11.7594173Z pass 2025-10-10T01:04:11.7594820Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:04:11.7595704Z TIMING: _recursive_pre_grad_passes:0.0133 _recursive_joint_graph_passes:0.83993 _recursive_post_grad_passes:0.27734 async_compile.wait:0.7425 code_gen:12.3217 inductor_compile:15.81472 backend_compile:21.74426 gc:0.00052 entire_frame_compile:26.65867 total_wall_time:26.65867 2025-10-10T01:04:11.7596830Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:17166 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:4835 2025-10-10T01:04:11.7597382Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-10-10T01:04:14.9907391Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:04:14.9908384Z import pynvml # type: ignore[import] 2025-10-10T01:04:18.4598990Z 2025-10-10T01:04:21.6788268Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:04:21.6792548Z loading model: 0it [00:03, ?it/s] 2025-10-10T01:04:21.6854233Z cpu eval XLNetLMHeadModel 2025-10-10T01:04:24.3706125Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:04:25.3527942Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:04:26.3235735Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:04:48.1862474Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1863133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1866467Z res = mod(**inputs) 2025-10-10T01:04:48.1872535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1877868Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1883016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-10-10T01:04:48.1887709Z word_emb_k = self.word_embedding(input_ids) 2025-10-10T01:04:48.1889439Z 2025-10-10T01:04:48.1889603Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1890027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1890447Z res = mod(**inputs) 2025-10-10T01:04:48.1891105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1891534Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1891938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-10-10T01:04:48.1892392Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-10-10T01:04:48.1892889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-10-10T01:04:48.1893385Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-10-10T01:04:48.1893875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-10-10T01:04:48.1894357Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-10-10T01:04:48.1894640Z 2025-10-10T01:04:48.1894810Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1895202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1895537Z res = mod(**inputs) 2025-10-10T01:04:48.1895917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1896326Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1896744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-10-10T01:04:48.1897198Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-10-10T01:04:48.1897692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-10-10T01:04:48.1898193Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-10-10T01:04:48.1898663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-10-10T01:04:48.1899162Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-10-10T01:04:48.1899376Z 2025-10-10T01:04:48.1899484Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1899848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1900174Z res = mod(**inputs) 2025-10-10T01:04:48.1900530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1900926Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1901321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1901707Z outputs = layer_module( 2025-10-10T01:04:48.1902091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1902499Z outputs = self.rel_attn( 2025-10-10T01:04:48.1902911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1903379Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1903814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1904271Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1904453Z 2025-10-10T01:04:48.1904563Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1904962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1905308Z res = mod(**inputs) 2025-10-10T01:04:48.1905742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1906366Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1906848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1907297Z outputs = layer_module( 2025-10-10T01:04:48.1907697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1908089Z outputs = self.rel_attn( 2025-10-10T01:04:48.1908472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1908972Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1909436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1910001Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1910185Z 2025-10-10T01:04:48.1910303Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1910693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1911048Z res = mod(**inputs) 2025-10-10T01:04:48.1911435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1911848Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1912250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1912667Z outputs = layer_module( 2025-10-10T01:04:48.1913064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1913495Z outputs = self.rel_attn( 2025-10-10T01:04:48.1913898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1914333Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1914760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1915235Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1915414Z 2025-10-10T01:04:48.1915533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1915913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1916275Z res = mod(**inputs) 2025-10-10T01:04:48.1916660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1917090Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1917525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1917944Z outputs = layer_module( 2025-10-10T01:04:48.1918340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1918783Z outputs = self.rel_attn( 2025-10-10T01:04:48.1919168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1919592Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1920032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1920517Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1920704Z 2025-10-10T01:04:48.1920821Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1921242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1921601Z res = mod(**inputs) 2025-10-10T01:04:48.1921988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1922418Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1922844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1923272Z outputs = layer_module( 2025-10-10T01:04:48.1923669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1924095Z outputs = self.rel_attn( 2025-10-10T01:04:48.1924496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1924971Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1925424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1925895Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1926083Z 2025-10-10T01:04:48.1926198Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1926791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1927147Z res = mod(**inputs) 2025-10-10T01:04:48.1927620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1928074Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1928505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1928952Z outputs = layer_module( 2025-10-10T01:04:48.1929356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1929817Z outputs = self.rel_attn( 2025-10-10T01:04:48.1930228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1930670Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1931128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1931764Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1931948Z 2025-10-10T01:04:48.1932064Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1932456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1932813Z res = mod(**inputs) 2025-10-10T01:04:48.1933781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1934219Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1934703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1935119Z outputs = layer_module( 2025-10-10T01:04:48.1935565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1935981Z outputs = self.rel_attn( 2025-10-10T01:04:48.1936379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1936802Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1937258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1937774Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1937965Z 2025-10-10T01:04:48.1938082Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1938460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1938793Z res = mod(**inputs) 2025-10-10T01:04:48.1939192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1939622Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1940051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1940467Z outputs = layer_module( 2025-10-10T01:04:48.1940917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1941362Z outputs = self.rel_attn( 2025-10-10T01:04:48.1941761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1942193Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1942643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1943128Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1943318Z 2025-10-10T01:04:48.1943435Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1943840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1944204Z res = mod(**inputs) 2025-10-10T01:04:48.1944615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1945046Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1945482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1945909Z outputs = layer_module( 2025-10-10T01:04:48.1946375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1946801Z outputs = self.rel_attn( 2025-10-10T01:04:48.1947213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1947657Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1948092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1948526Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1948701Z 2025-10-10T01:04:48.1948809Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1949183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1949516Z res = mod(**inputs) 2025-10-10T01:04:48.1949926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1950332Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1950741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1951126Z outputs = layer_module( 2025-10-10T01:04:48.1951491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1951875Z outputs = self.rel_attn( 2025-10-10T01:04:48.1952239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1952661Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1953076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1953517Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1953686Z 2025-10-10T01:04:48.1953800Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1954165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1954495Z res = mod(**inputs) 2025-10-10T01:04:48.1954863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1955267Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1955686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1956101Z outputs = layer_module( 2025-10-10T01:04:48.1956467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1956848Z outputs = self.rel_attn( 2025-10-10T01:04:48.1957215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1957611Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1958033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1958471Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1958639Z 2025-10-10T01:04:48.1958759Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1959127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1959441Z res = mod(**inputs) 2025-10-10T01:04:48.1959805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1960205Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1960600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1960979Z outputs = layer_module( 2025-10-10T01:04:48.1961343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1961734Z outputs = self.rel_attn( 2025-10-10T01:04:48.1962110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1962519Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1962992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1963423Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1963588Z 2025-10-10T01:04:48.1963692Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1964056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1964365Z res = mod(**inputs) 2025-10-10T01:04:48.1964708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1965091Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1965472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1965842Z outputs = layer_module( 2025-10-10T01:04:48.1966197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1966600Z outputs = self.rel_attn( 2025-10-10T01:04:48.1966966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1967365Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1967781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1968226Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1968405Z 2025-10-10T01:04:48.1968512Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1968881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1969221Z res = mod(**inputs) 2025-10-10T01:04:48.1969599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1970012Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1970407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1970797Z outputs = layer_module( 2025-10-10T01:04:48.1971180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1971558Z outputs = self.rel_attn( 2025-10-10T01:04:48.1971927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1972330Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1972760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1973217Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1973390Z 2025-10-10T01:04:48.1973499Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1973869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1974206Z res = mod(**inputs) 2025-10-10T01:04:48.1974586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1974984Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1975374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1975761Z outputs = layer_module( 2025-10-10T01:04:48.1976131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1976514Z outputs = self.rel_attn( 2025-10-10T01:04:48.1976881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1977295Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1977729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1978197Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1978366Z 2025-10-10T01:04:48.1978480Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1978848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1979168Z res = mod(**inputs) 2025-10-10T01:04:48.1979529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1979930Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1980356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1980742Z outputs = layer_module( 2025-10-10T01:04:48.1981116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1981506Z outputs = self.rel_attn( 2025-10-10T01:04:48.1981883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1982288Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1982720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1983175Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1983352Z 2025-10-10T01:04:48.1983460Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1983855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1984204Z res = mod(**inputs) 2025-10-10T01:04:48.1984598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1985035Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1985468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1985882Z outputs = layer_module( 2025-10-10T01:04:48.1986371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1986818Z outputs = self.rel_attn( 2025-10-10T01:04:48.1987242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1987677Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1988107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1988562Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1988742Z 2025-10-10T01:04:48.1988854Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1989229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1989564Z res = mod(**inputs) 2025-10-10T01:04:48.1989930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1990341Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1990747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1991135Z outputs = layer_module( 2025-10-10T01:04:48.1991516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1991900Z outputs = self.rel_attn( 2025-10-10T01:04:48.1992279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1992717Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1993149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1993604Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1993773Z 2025-10-10T01:04:48.1993881Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1994252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1994583Z res = mod(**inputs) 2025-10-10T01:04:48.1994977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.1995377Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.1995781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.1996177Z outputs = layer_module( 2025-10-10T01:04:48.1996552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.1996940Z outputs = self.rel_attn( 2025-10-10T01:04:48.1997328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.1997759Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.1998210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.1998742Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.1998922Z 2025-10-10T01:04:48.1999050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.1999406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.1999741Z res = mod(**inputs) 2025-10-10T01:04:48.2000105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2000508Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2000902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2001294Z outputs = layer_module( 2025-10-10T01:04:48.2001666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2002052Z outputs = self.rel_attn( 2025-10-10T01:04:48.2002429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2002836Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2003279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2003732Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2003900Z 2025-10-10T01:04:48.2004015Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2004380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2004700Z res = mod(**inputs) 2025-10-10T01:04:48.2005066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2005469Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2005877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2006258Z outputs = layer_module( 2025-10-10T01:04:48.2006629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2007043Z outputs = self.rel_attn( 2025-10-10T01:04:48.2007422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2007833Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2008260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2008712Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2008887Z 2025-10-10T01:04:48.2008996Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2009388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2009721Z res = mod(**inputs) 2025-10-10T01:04:48.2010083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2010489Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2010890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2011280Z outputs = layer_module( 2025-10-10T01:04:48.2011655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2012033Z outputs = self.rel_attn( 2025-10-10T01:04:48.2012402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2012832Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2013251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2013687Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2013863Z 2025-10-10T01:04:48.2013970Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2014341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2014661Z res = mod(**inputs) 2025-10-10T01:04:48.2015027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2015424Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2015825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2016215Z outputs = layer_module( 2025-10-10T01:04:48.2016590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2016973Z outputs = self.rel_attn( 2025-10-10T01:04:48.2017346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2017752Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2018180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2018643Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2018821Z 2025-10-10T01:04:48.2018934Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2019321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2019672Z res = mod(**inputs) 2025-10-10T01:04:48.2020066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2020497Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2020915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2021354Z outputs = layer_module( 2025-10-10T01:04:48.2021752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2022170Z outputs = self.rel_attn( 2025-10-10T01:04:48.2022568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2023005Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2023457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2023958Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2024137Z 2025-10-10T01:04:48.2024258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2024640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2024992Z res = mod(**inputs) 2025-10-10T01:04:48.2025379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2025809Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2026312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2026729Z outputs = layer_module( 2025-10-10T01:04:48.2027124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2027585Z outputs = self.rel_attn( 2025-10-10T01:04:48.2027990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2028399Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2028567Z 2025-10-10T01:04:48.2028676Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2029051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2029399Z res = mod(**inputs) 2025-10-10T01:04:48.2029787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2030213Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2030646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2031130Z outputs = layer_module( 2025-10-10T01:04:48.2031676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2032100Z outputs = self.rel_attn( 2025-10-10T01:04:48.2032499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2032956Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2033132Z 2025-10-10T01:04:48.2033246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2033637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2033982Z res = mod(**inputs) 2025-10-10T01:04:48.2034373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2034806Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2035247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2035662Z outputs = layer_module( 2025-10-10T01:04:48.2036051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2036509Z outputs = self.rel_attn( 2025-10-10T01:04:48.2036908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2037331Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2037763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2038218Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2038414Z 2025-10-10T01:04:48.2038522Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2038885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2039237Z res = mod(**inputs) 2025-10-10T01:04:48.2039593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2039993Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2040389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-10-10T01:04:48.2040828Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-10-10T01:04:48.2041313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-10-10T01:04:48.2041794Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-10-10T01:04:48.2042290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-10-10T01:04:48.2042826Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-10-10T01:04:48.2043041Z 2025-10-10T01:04:48.2043145Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2043516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2043829Z res = mod(**inputs) 2025-10-10T01:04:48.2044188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2044585Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2044980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2045364Z outputs = layer_module( 2025-10-10T01:04:48.2045725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2046107Z outputs = self.rel_attn( 2025-10-10T01:04:48.2046474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2046919Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2047106Z 2025-10-10T01:04:48.2047218Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2047565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2047885Z res = mod(**inputs) 2025-10-10T01:04:48.2048246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2048643Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2049025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2049407Z outputs = layer_module( 2025-10-10T01:04:48.2049771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2050150Z outputs = self.rel_attn( 2025-10-10T01:04:48.2050527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2050914Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2051303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2051744Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2051923Z 2025-10-10T01:04:48.2052032Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2052375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2052682Z res = mod(**inputs) 2025-10-10T01:04:48.2053053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2053447Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2053842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2054217Z outputs = layer_module( 2025-10-10T01:04:48.2054586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2054969Z outputs = self.rel_attn( 2025-10-10T01:04:48.2055325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2055727Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2055878Z 2025-10-10T01:04:48.2056010Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2056378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2056692Z res = mod(**inputs) 2025-10-10T01:04:48.2057042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2057425Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2057801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2058169Z outputs = layer_module( 2025-10-10T01:04:48.2058521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2058894Z outputs = self.rel_attn( 2025-10-10T01:04:48.2059281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2059669Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2060065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2060513Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2060689Z 2025-10-10T01:04:48.2060802Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2061152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2061474Z res = mod(**inputs) 2025-10-10T01:04:48.2061831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2062227Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2062619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2062996Z outputs = layer_module( 2025-10-10T01:04:48.2063359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2063744Z outputs = self.rel_attn( 2025-10-10T01:04:48.2064149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2064557Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2064989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2065451Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2065830Z 2025-10-10T01:04:48.2065956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2066408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2066776Z res = mod(**inputs) 2025-10-10T01:04:48.2067235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2067686Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2068117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2068537Z outputs = layer_module( 2025-10-10T01:04:48.2068927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2069387Z outputs = self.rel_attn( 2025-10-10T01:04:48.2069788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2070223Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2070686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2071189Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2071377Z 2025-10-10T01:04:48.2071490Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2071879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2072224Z res = mod(**inputs) 2025-10-10T01:04:48.2072604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2073032Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2073455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2073865Z outputs = layer_module( 2025-10-10T01:04:48.2074260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2074818Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2075373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2075773Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2076165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2076540Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2076901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2077274Z output = self.layer_1(output) 2025-10-10T01:04:48.2077398Z 2025-10-10T01:04:48.2077509Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2077856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2078165Z res = mod(**inputs) 2025-10-10T01:04:48.2078524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2078921Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2079338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2079726Z outputs = layer_module( 2025-10-10T01:04:48.2080074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2080579Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2081088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2081480Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2081910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2082278Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2082647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2083036Z output = self.activation_function(output) 2025-10-10T01:04:48.2083389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2083724Z return self.act(input) 2025-10-10T01:04:48.2083845Z 2025-10-10T01:04:48.2083949Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2084307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2084638Z res = mod(**inputs) 2025-10-10T01:04:48.2085034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2085411Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2085793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2086173Z outputs = layer_module( 2025-10-10T01:04:48.2086539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2087055Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2087565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2087960Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2088349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2088735Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2089104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2089488Z output = self.layer_2(output) 2025-10-10T01:04:48.2089619Z 2025-10-10T01:04:48.2089725Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2090085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2090407Z res = mod(**inputs) 2025-10-10T01:04:48.2090765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2091170Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2091576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2091979Z outputs = layer_module( 2025-10-10T01:04:48.2092348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2092726Z outputs = self.rel_attn( 2025-10-10T01:04:48.2093111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2093521Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2093673Z 2025-10-10T01:04:48.2093785Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2094133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2094454Z res = mod(**inputs) 2025-10-10T01:04:48.2094812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2095206Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2095617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2095991Z outputs = layer_module( 2025-10-10T01:04:48.2096354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2096741Z outputs = self.rel_attn( 2025-10-10T01:04:48.2097121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2097545Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2097703Z 2025-10-10T01:04:48.2097811Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2098176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2098506Z res = mod(**inputs) 2025-10-10T01:04:48.2098909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2099298Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2099689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2100069Z outputs = layer_module( 2025-10-10T01:04:48.2100432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2100822Z outputs = self.rel_attn( 2025-10-10T01:04:48.2101189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2101586Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2101996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2102473Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2102665Z 2025-10-10T01:04:48.2102779Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2103136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2103466Z res = mod(**inputs) 2025-10-10T01:04:48.2103853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2104282Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2104705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2105115Z outputs = layer_module( 2025-10-10T01:04:48.2105509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2105923Z outputs = self.rel_attn( 2025-10-10T01:04:48.2106394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2106878Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2107108Z 2025-10-10T01:04:48.2107223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2107614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2107987Z res = mod(**inputs) 2025-10-10T01:04:48.2108380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2108855Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2109288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2109703Z outputs = layer_module( 2025-10-10T01:04:48.2110135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2110543Z outputs = self.rel_attn( 2025-10-10T01:04:48.2110945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2111367Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2111801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2112301Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2112498Z 2025-10-10T01:04:48.2112612Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2113001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2113351Z res = mod(**inputs) 2025-10-10T01:04:48.2113785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2114229Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2114650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2115066Z outputs = layer_module( 2025-10-10T01:04:48.2115460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2115874Z outputs = self.rel_attn( 2025-10-10T01:04:48.2116266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2116712Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2116884Z 2025-10-10T01:04:48.2116998Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2117391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2117740Z res = mod(**inputs) 2025-10-10T01:04:48.2118114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2118521Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2118925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2119315Z outputs = layer_module( 2025-10-10T01:04:48.2119688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2120067Z outputs = self.rel_attn( 2025-10-10T01:04:48.2120441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2120835Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2121246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2121715Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2121913Z 2025-10-10T01:04:48.2122049Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2122432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2122777Z res = mod(**inputs) 2025-10-10T01:04:48.2123161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2123582Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2124007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2124422Z outputs = layer_module( 2025-10-10T01:04:48.2124838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2125252Z outputs = self.rel_attn( 2025-10-10T01:04:48.2125642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2126082Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2126539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2127024Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2127206Z 2025-10-10T01:04:48.2127326Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2127707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2128056Z res = mod(**inputs) 2025-10-10T01:04:48.2128467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2128920Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2129355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2129780Z outputs = layer_module( 2025-10-10T01:04:48.2130180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2130598Z outputs = self.rel_attn( 2025-10-10T01:04:48.2131012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2131420Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2132007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2132501Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2132688Z 2025-10-10T01:04:48.2132819Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2133192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2133521Z res = mod(**inputs) 2025-10-10T01:04:48.2133896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2134311Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2134717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2135107Z outputs = layer_module( 2025-10-10T01:04:48.2135485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2136023Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2136569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2136982Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2137413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2137811Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2138206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2138607Z output = self.layer_1(output) 2025-10-10T01:04:48.2138736Z 2025-10-10T01:04:48.2138850Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2139210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2142319Z res = mod(**inputs) 2025-10-10T01:04:48.2142711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2143127Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2143529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2143937Z outputs = layer_module( 2025-10-10T01:04:48.2144341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2144905Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2145471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2145910Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2146512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2146930Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2147342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2147776Z output = self.activation_function(output) 2025-10-10T01:04:48.2148133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2148477Z return self.act(input) 2025-10-10T01:04:48.2148588Z 2025-10-10T01:04:48.2148700Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2149059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2149398Z res = mod(**inputs) 2025-10-10T01:04:48.2149781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2150202Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2150606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2151012Z outputs = layer_module( 2025-10-10T01:04:48.2151378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2151887Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2152415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2152814Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2153207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2153604Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2153994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2154375Z output = self.layer_2(output) 2025-10-10T01:04:48.2154522Z 2025-10-10T01:04:48.2154629Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2154993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2155315Z res = mod(**inputs) 2025-10-10T01:04:48.2155675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2156076Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2156466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2156846Z outputs = layer_module( 2025-10-10T01:04:48.2157278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2157663Z outputs = self.rel_attn( 2025-10-10T01:04:48.2158025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2158437Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2158596Z 2025-10-10T01:04:48.2158702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2159058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2159380Z res = mod(**inputs) 2025-10-10T01:04:48.2159732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2160126Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2160542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2160944Z outputs = layer_module( 2025-10-10T01:04:48.2161316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2161699Z outputs = self.rel_attn( 2025-10-10T01:04:48.2162075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2162498Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2162655Z 2025-10-10T01:04:48.2162770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2163127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2163455Z res = mod(**inputs) 2025-10-10T01:04:48.2163827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2164269Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2164671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2165058Z outputs = layer_module( 2025-10-10T01:04:48.2165431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2165820Z outputs = self.rel_attn( 2025-10-10T01:04:48.2166196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2166593Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2166995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2167462Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2167664Z 2025-10-10T01:04:48.2167773Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2168138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2168460Z res = mod(**inputs) 2025-10-10T01:04:48.2168853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2169249Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2169644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2170026Z outputs = layer_module( 2025-10-10T01:04:48.2170381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2170760Z outputs = self.rel_attn( 2025-10-10T01:04:48.2171159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2171613Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2171800Z 2025-10-10T01:04:48.2171911Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2172263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2172588Z res = mod(**inputs) 2025-10-10T01:04:48.2172947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2173382Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2173768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2174148Z outputs = layer_module( 2025-10-10T01:04:48.2174578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2174989Z outputs = self.rel_attn( 2025-10-10T01:04:48.2175365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2175756Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2176163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2176632Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2176810Z 2025-10-10T01:04:48.2176933Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2177281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2177583Z res = mod(**inputs) 2025-10-10T01:04:48.2177942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2178335Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2178726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2179099Z outputs = layer_module( 2025-10-10T01:04:48.2179461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2179841Z outputs = self.rel_attn( 2025-10-10T01:04:48.2180207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2180624Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2180782Z 2025-10-10T01:04:48.2180888Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2181256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2181588Z res = mod(**inputs) 2025-10-10T01:04:48.2181956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2182363Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2182780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2183171Z outputs = layer_module( 2025-10-10T01:04:48.2183546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2183941Z outputs = self.rel_attn( 2025-10-10T01:04:48.2184315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2184737Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2185195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2185690Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2185878Z 2025-10-10T01:04:48.2185999Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2186464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2186812Z res = mod(**inputs) 2025-10-10T01:04:48.2187209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2187633Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2188031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2188420Z outputs = layer_module( 2025-10-10T01:04:48.2188814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2189210Z outputs = self.rel_attn( 2025-10-10T01:04:48.2189577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2189984Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2190402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2190850Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2191023Z 2025-10-10T01:04:48.2191129Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2191484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2191799Z res = mod(**inputs) 2025-10-10T01:04:48.2192163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2192562Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2192957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2193342Z outputs = layer_module( 2025-10-10T01:04:48.2193703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2194084Z outputs = self.rel_attn( 2025-10-10T01:04:48.2194458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2194868Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2195298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2195744Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2195922Z 2025-10-10T01:04:48.2196031Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2196394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2196722Z res = mod(**inputs) 2025-10-10T01:04:48.2197103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2197513Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2197928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2198311Z outputs = layer_module( 2025-10-10T01:04:48.2198675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2199193Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2199754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2200165Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2200569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2200977Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2201346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2201733Z output = self.layer_1(output) 2025-10-10T01:04:48.2201865Z 2025-10-10T01:04:48.2201970Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2202337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2202667Z res = mod(**inputs) 2025-10-10T01:04:48.2203057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2203481Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2203883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2204276Z outputs = layer_module( 2025-10-10T01:04:48.2204646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2205172Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2205733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2206162Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2206585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2206995Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2207396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2207809Z output = self.activation_function(output) 2025-10-10T01:04:48.2208179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2208534Z return self.act(input) 2025-10-10T01:04:48.2208649Z 2025-10-10T01:04:48.2208756Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2209121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2209451Z res = mod(**inputs) 2025-10-10T01:04:48.2209821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2210224Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2210631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2211023Z outputs = layer_module( 2025-10-10T01:04:48.2211399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2211943Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2212466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2212876Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2213308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2213731Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2214159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2214552Z output = self.layer_2(output) 2025-10-10T01:04:48.2214688Z 2025-10-10T01:04:48.2214799Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2215177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2215528Z res = mod(**inputs) 2025-10-10T01:04:48.2215913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2216346Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2216776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2217194Z outputs = layer_module( 2025-10-10T01:04:48.2217619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2218032Z outputs = self.rel_attn( 2025-10-10T01:04:48.2218412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2218833Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2218990Z 2025-10-10T01:04:48.2219105Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2219475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2219797Z res = mod(**inputs) 2025-10-10T01:04:48.2220178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2220607Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2221039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2221447Z outputs = layer_module( 2025-10-10T01:04:48.2221845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2222257Z outputs = self.rel_attn( 2025-10-10T01:04:48.2222669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2223128Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2223301Z 2025-10-10T01:04:48.2223421Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2223819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2224190Z res = mod(**inputs) 2025-10-10T01:04:48.2224593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2225025Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2225444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2225857Z outputs = layer_module( 2025-10-10T01:04:48.2226328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2226777Z outputs = self.rel_attn( 2025-10-10T01:04:48.2227171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2227597Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2228039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2228549Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2228756Z 2025-10-10T01:04:48.2228912Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2229315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2229666Z res = mod(**inputs) 2025-10-10T01:04:48.2230061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2230497Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2230929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2231344Z outputs = layer_module( 2025-10-10T01:04:48.2231891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2232313Z outputs = self.rel_attn( 2025-10-10T01:04:48.2232753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2233293Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2233503Z 2025-10-10T01:04:48.2233619Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2234013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2234324Z res = mod(**inputs) 2025-10-10T01:04:48.2234670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2235051Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2235435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2235810Z outputs = layer_module( 2025-10-10T01:04:48.2236168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2236550Z outputs = self.rel_attn( 2025-10-10T01:04:48.2236903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2237287Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2237686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2238154Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2238339Z 2025-10-10T01:04:48.2238462Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2238844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2239197Z res = mod(**inputs) 2025-10-10T01:04:48.2239587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2240017Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2240418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2240788Z outputs = layer_module( 2025-10-10T01:04:48.2241146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2241538Z outputs = self.rel_attn( 2025-10-10T01:04:48.2241912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2242327Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2242493Z 2025-10-10T01:04:48.2242598Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2242962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2243290Z res = mod(**inputs) 2025-10-10T01:04:48.2243690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2244091Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2244488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2244863Z outputs = layer_module( 2025-10-10T01:04:48.2245221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2245586Z outputs = self.rel_attn( 2025-10-10T01:04:48.2245948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2246330Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2246720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2247197Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2247376Z 2025-10-10T01:04:48.2247485Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2247856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2248196Z res = mod(**inputs) 2025-10-10T01:04:48.2248571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2248987Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2249385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2249777Z outputs = layer_module( 2025-10-10T01:04:48.2250158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2250559Z outputs = self.rel_attn( 2025-10-10T01:04:48.2250938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2251360Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2251798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2252258Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2252431Z 2025-10-10T01:04:48.2252548Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2252916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2253255Z res = mod(**inputs) 2025-10-10T01:04:48.2253631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2254045Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2254459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2254861Z outputs = layer_module( 2025-10-10T01:04:48.2255240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2256612Z outputs = self.rel_attn( 2025-10-10T01:04:48.2256993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2257399Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2257833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2258289Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2258460Z 2025-10-10T01:04:48.2258602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2258973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2259296Z res = mod(**inputs) 2025-10-10T01:04:48.2259665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2260074Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2260477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2260911Z outputs = layer_module( 2025-10-10T01:04:48.2261315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2261879Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2262471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2262921Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2263338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2263757Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2264167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2264581Z output = self.layer_1(output) 2025-10-10T01:04:48.2264716Z 2025-10-10T01:04:48.2264840Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2265225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2265574Z res = mod(**inputs) 2025-10-10T01:04:48.2265963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2266476Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2266915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2267328Z outputs = layer_module( 2025-10-10T01:04:48.2267734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2268304Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2268877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2269313Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2269727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2270154Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2270563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2270998Z output = self.activation_function(output) 2025-10-10T01:04:48.2271414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2271793Z return self.act(input) 2025-10-10T01:04:48.2271924Z 2025-10-10T01:04:48.2272041Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2272440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2272803Z res = mod(**inputs) 2025-10-10T01:04:48.2273192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2273632Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2274090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2274508Z outputs = layer_module( 2025-10-10T01:04:48.2274901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2275467Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2276038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2276454Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2276844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2277229Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2277629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2278025Z output = self.layer_2(output) 2025-10-10T01:04:48.2278147Z 2025-10-10T01:04:48.2278258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2278608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2278928Z res = mod(**inputs) 2025-10-10T01:04:48.2279291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2279691Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2280089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2280480Z outputs = layer_module( 2025-10-10T01:04:48.2280829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2281206Z outputs = self.rel_attn( 2025-10-10T01:04:48.2281565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2281963Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2282112Z 2025-10-10T01:04:48.2282215Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2282564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2282880Z res = mod(**inputs) 2025-10-10T01:04:48.2283228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2283612Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2283989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2284359Z outputs = layer_module( 2025-10-10T01:04:48.2284722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2285101Z outputs = self.rel_attn( 2025-10-10T01:04:48.2285464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2285899Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2286058Z 2025-10-10T01:04:48.2286163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2286525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2286838Z res = mod(**inputs) 2025-10-10T01:04:48.2287180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2287564Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2287975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2288351Z outputs = layer_module( 2025-10-10T01:04:48.2288711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2289081Z outputs = self.rel_attn( 2025-10-10T01:04:48.2289457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2289830Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2290216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2290662Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2290841Z 2025-10-10T01:04:48.2290942Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2291321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2291634Z res = mod(**inputs) 2025-10-10T01:04:48.2291982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2292361Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2292744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2293116Z outputs = layer_module( 2025-10-10T01:04:48.2293471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2293837Z outputs = self.rel_attn( 2025-10-10T01:04:48.2294188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2294622Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2294816Z 2025-10-10T01:04:48.2294922Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2295279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2295597Z res = mod(**inputs) 2025-10-10T01:04:48.2295851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2295936Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2296192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2296262Z outputs = layer_module( 2025-10-10T01:04:48.2296517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2296587Z outputs = self.rel_attn( 2025-10-10T01:04:48.2296839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2296921Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2297186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2297360Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2297364Z 2025-10-10T01:04:48.2297465Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2297666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2297734Z res = mod(**inputs) 2025-10-10T01:04:48.2297982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2298070Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2298338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2298415Z outputs = layer_module( 2025-10-10T01:04:48.2298664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2298734Z outputs = self.rel_attn( 2025-10-10T01:04:48.2298990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2299092Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2299095Z 2025-10-10T01:04:48.2299205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2299403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2299467Z res = mod(**inputs) 2025-10-10T01:04:48.2299741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2299841Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2300094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2300167Z outputs = layer_module( 2025-10-10T01:04:48.2300431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2300506Z outputs = self.rel_attn( 2025-10-10T01:04:48.2300763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2300851Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2301129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2301269Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2301274Z 2025-10-10T01:04:48.2301386Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2301600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2301682Z res = mod(**inputs) 2025-10-10T01:04:48.2301962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2302064Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2302337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2302420Z outputs = layer_module( 2025-10-10T01:04:48.2302695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2302771Z outputs = self.rel_attn( 2025-10-10T01:04:48.2303054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2303159Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2303463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2303607Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2303611Z 2025-10-10T01:04:48.2303720Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2303939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2304010Z res = mod(**inputs) 2025-10-10T01:04:48.2304290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2304380Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2304670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2304756Z outputs = layer_module( 2025-10-10T01:04:48.2305025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2305107Z outputs = self.rel_attn( 2025-10-10T01:04:48.2305375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2305479Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2305769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2305890Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2305894Z 2025-10-10T01:04:48.2306014Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2306325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2306432Z res = mod(**inputs) 2025-10-10T01:04:48.2306717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2306813Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2307104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2307182Z outputs = layer_module( 2025-10-10T01:04:48.2307470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2307699Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2307975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2308062Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2308322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2308408Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2308668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2308754Z output = self.layer_1(output) 2025-10-10T01:04:48.2308758Z 2025-10-10T01:04:48.2308865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2309077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2309150Z res = mod(**inputs) 2025-10-10T01:04:48.2309403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2309493Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2309758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2309836Z outputs = layer_module( 2025-10-10T01:04:48.2310092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2310319Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2310593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2310676Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2310955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2311036Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2311337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2311443Z output = self.activation_function(output) 2025-10-10T01:04:48.2311673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2311760Z return self.act(input) 2025-10-10T01:04:48.2311764Z 2025-10-10T01:04:48.2311874Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2312092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2312163Z res = mod(**inputs) 2025-10-10T01:04:48.2312438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2312545Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2312821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2312915Z outputs = layer_module( 2025-10-10T01:04:48.2313171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2313378Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2313653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2313734Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2313998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2314073Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2314338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2314423Z output = self.layer_2(output) 2025-10-10T01:04:48.2314427Z 2025-10-10T01:04:48.2314539Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2314762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2314834Z res = mod(**inputs) 2025-10-10T01:04:48.2315112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2315202Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2315475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2315558Z outputs = layer_module( 2025-10-10T01:04:48.2315830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2315914Z outputs = self.rel_attn( 2025-10-10T01:04:48.2316189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2316296Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2316310Z 2025-10-10T01:04:48.2316422Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2316655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2316733Z res = mod(**inputs) 2025-10-10T01:04:48.2317005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2317101Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2317371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2317443Z outputs = layer_module( 2025-10-10T01:04:48.2317743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2317822Z outputs = self.rel_attn( 2025-10-10T01:04:48.2318098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2318211Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2318215Z 2025-10-10T01:04:48.2318326Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2318544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2318613Z res = mod(**inputs) 2025-10-10T01:04:48.2318894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2318982Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2319279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2319375Z outputs = layer_module( 2025-10-10T01:04:48.2319646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2319729Z outputs = self.rel_attn( 2025-10-10T01:04:48.2320002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2320091Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2320382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2320525Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2320529Z 2025-10-10T01:04:48.2320645Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2320858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2320938Z res = mod(**inputs) 2025-10-10T01:04:48.2321213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2321308Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2321581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2321654Z outputs = layer_module( 2025-10-10T01:04:48.2321934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2322008Z outputs = self.rel_attn( 2025-10-10T01:04:48.2322288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2322433Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2322439Z 2025-10-10T01:04:48.2322552Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2322772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2322841Z res = mod(**inputs) 2025-10-10T01:04:48.2323139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2323228Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2323502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2323582Z outputs = layer_module( 2025-10-10T01:04:48.2323851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2323932Z outputs = self.rel_attn( 2025-10-10T01:04:48.2324221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2324314Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2324604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2324747Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2324751Z 2025-10-10T01:04:48.2324870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2325080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2325158Z res = mod(**inputs) 2025-10-10T01:04:48.2325431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2325515Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2325796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2325879Z outputs = layer_module( 2025-10-10T01:04:48.2326145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2326217Z outputs = self.rel_attn( 2025-10-10T01:04:48.2326482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2326588Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2326592Z 2025-10-10T01:04:48.2326698Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2326907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2326975Z res = mod(**inputs) 2025-10-10T01:04:48.2327241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2327330Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2327587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2327665Z outputs = layer_module( 2025-10-10T01:04:48.2327922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2328002Z outputs = self.rel_attn( 2025-10-10T01:04:48.2328266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2328340Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2328619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2328746Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2328749Z 2025-10-10T01:04:48.2328866Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2329065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2329140Z res = mod(**inputs) 2025-10-10T01:04:48.2329397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2329502Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2329768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2329837Z outputs = layer_module( 2025-10-10T01:04:48.2330103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2330174Z outputs = self.rel_attn( 2025-10-10T01:04:48.2330449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2330551Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2330827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2330950Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2330954Z 2025-10-10T01:04:48.2331062Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2331267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2331332Z res = mod(**inputs) 2025-10-10T01:04:48.2331729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2331828Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2332089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2332227Z outputs = layer_module( 2025-10-10T01:04:48.2332481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2332553Z outputs = self.rel_attn( 2025-10-10T01:04:48.2332816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2332905Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2333188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2333301Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2333305Z 2025-10-10T01:04:48.2333418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2333618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2333689Z res = mod(**inputs) 2025-10-10T01:04:48.2333950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2334034Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2334297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2334367Z outputs = layer_module( 2025-10-10T01:04:48.2334618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2334835Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2335098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2335186Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2335445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2335524Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2335778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2335885Z output = self.layer_1(output) 2025-10-10T01:04:48.2335889Z 2025-10-10T01:04:48.2336001Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2336202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2336276Z res = mod(**inputs) 2025-10-10T01:04:48.2336540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2336624Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2336946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2337019Z outputs = layer_module( 2025-10-10T01:04:48.2337285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2337495Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2337770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2337851Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2338111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2338193Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2338468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2338582Z output = self.activation_function(output) 2025-10-10T01:04:48.2338802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2338874Z return self.act(input) 2025-10-10T01:04:48.2338879Z 2025-10-10T01:04:48.2338993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2339192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2339264Z res = mod(**inputs) 2025-10-10T01:04:48.2339519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2339604Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2339869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2339944Z outputs = layer_module( 2025-10-10T01:04:48.2340209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2340419Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2340676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2340753Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2340998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2341074Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2341313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2341392Z output = self.layer_2(output) 2025-10-10T01:04:48.2341396Z 2025-10-10T01:04:48.2341499Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2341689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2341760Z res = mod(**inputs) 2025-10-10T01:04:48.2342017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2342105Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2342346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2342418Z outputs = layer_module( 2025-10-10T01:04:48.2342657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2342725Z outputs = self.rel_attn( 2025-10-10T01:04:48.2342990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2343087Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2343091Z 2025-10-10T01:04:48.2343196Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2343386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2343452Z res = mod(**inputs) 2025-10-10T01:04:48.2343708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2343789Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2344041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2344109Z outputs = layer_module( 2025-10-10T01:04:48.2344373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2344476Z outputs = self.rel_attn( 2025-10-10T01:04:48.2344717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2344822Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2344827Z 2025-10-10T01:04:48.2344928Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2345129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2345193Z res = mod(**inputs) 2025-10-10T01:04:48.2345439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2345527Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2345781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2345862Z outputs = layer_module( 2025-10-10T01:04:48.2346115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2346235Z outputs = self.rel_attn( 2025-10-10T01:04:48.2346505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2346584Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2346863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2346998Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2347002Z 2025-10-10T01:04:48.2347114Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2347310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2347377Z res = mod(**inputs) 2025-10-10T01:04:48.2347657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2347736Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2347988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2348076Z outputs = layer_module( 2025-10-10T01:04:48.2348320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2348395Z outputs = self.rel_attn( 2025-10-10T01:04:48.2348639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2348776Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2348779Z 2025-10-10T01:04:48.2348883Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2349101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2349168Z res = mod(**inputs) 2025-10-10T01:04:48.2349427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2349516Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2349757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2349831Z outputs = layer_module( 2025-10-10T01:04:48.2350072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2350139Z outputs = self.rel_attn( 2025-10-10T01:04:48.2350385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2350491Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2350759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2350882Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2350887Z 2025-10-10T01:04:48.2350993Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2351181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2351245Z res = mod(**inputs) 2025-10-10T01:04:48.2351499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2351576Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2351828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2351897Z outputs = layer_module( 2025-10-10T01:04:48.2352141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2352215Z outputs = self.rel_attn( 2025-10-10T01:04:48.2352461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2352565Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2352568Z 2025-10-10T01:04:48.2352668Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2352865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2352927Z res = mod(**inputs) 2025-10-10T01:04:48.2353174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2353259Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2353508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2353581Z outputs = layer_module( 2025-10-10T01:04:48.2353824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2353910Z outputs = self.rel_attn( 2025-10-10T01:04:48.2354159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2354232Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2354499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2354619Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2354622Z 2025-10-10T01:04:48.2354723Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2354939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2355005Z res = mod(**inputs) 2025-10-10T01:04:48.2355258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2355339Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2355590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2355657Z outputs = layer_module( 2025-10-10T01:04:48.2355902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2355976Z outputs = self.rel_attn( 2025-10-10T01:04:48.2356219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2356328Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2356616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2356724Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2356729Z 2025-10-10T01:04:48.2356837Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2357038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2357108Z res = mod(**inputs) 2025-10-10T01:04:48.2357347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2357431Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2357674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2357739Z outputs = layer_module( 2025-10-10T01:04:48.2357994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2358060Z outputs = self.rel_attn( 2025-10-10T01:04:48.2358311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2358399Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2358662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2358778Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2358782Z 2025-10-10T01:04:48.2358881Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2359078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2359140Z res = mod(**inputs) 2025-10-10T01:04:48.2359387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2359473Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2359716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2359809Z outputs = layer_module( 2025-10-10T01:04:48.2360050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2360253Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2360506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2360585Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2360853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2360928Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2361179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2361254Z output = self.layer_1(output) 2025-10-10T01:04:48.2361257Z 2025-10-10T01:04:48.2361364Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2361554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2361618Z res = mod(**inputs) 2025-10-10T01:04:48.2361873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2361950Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2362202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2362302Z outputs = layer_module( 2025-10-10T01:04:48.2362546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2362749Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2363000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2363082Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2363323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2363395Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2363640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2363730Z output = self.activation_function(output) 2025-10-10T01:04:48.2363942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2364011Z return self.act(input) 2025-10-10T01:04:48.2364014Z 2025-10-10T01:04:48.2364122Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2364308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2364372Z res = mod(**inputs) 2025-10-10T01:04:48.2364619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2364698Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2364945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2365010Z outputs = layer_module( 2025-10-10T01:04:48.2365252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2365455Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2365698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2365797Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2366039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2366116Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2366358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2366432Z output = self.layer_2(output) 2025-10-10T01:04:48.2366436Z 2025-10-10T01:04:48.2366544Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2366756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2366826Z res = mod(**inputs) 2025-10-10T01:04:48.2367070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2367150Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2367402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2367469Z outputs = layer_module( 2025-10-10T01:04:48.2367724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2367793Z outputs = self.rel_attn( 2025-10-10T01:04:48.2368045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2368176Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2368179Z 2025-10-10T01:04:48.2368280Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2368479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2368544Z res = mod(**inputs) 2025-10-10T01:04:48.2368794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2368874Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2369122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2369196Z outputs = layer_module( 2025-10-10T01:04:48.2369441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2369527Z outputs = self.rel_attn( 2025-10-10T01:04:48.2369770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2369866Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2369878Z 2025-10-10T01:04:48.2369977Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2370162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2370232Z res = mod(**inputs) 2025-10-10T01:04:48.2370471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2370557Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2370798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2370862Z outputs = layer_module( 2025-10-10T01:04:48.2371111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2371179Z outputs = self.rel_attn( 2025-10-10T01:04:48.2371424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2371511Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2371768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2371901Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2371904Z 2025-10-10T01:04:48.2371999Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2372193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2372256Z res = mod(**inputs) 2025-10-10T01:04:48.2372523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2372604Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2372843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2372916Z outputs = layer_module( 2025-10-10T01:04:48.2373152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2373224Z outputs = self.rel_attn( 2025-10-10T01:04:48.2373461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2373586Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2373589Z 2025-10-10T01:04:48.2373693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2373892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2373976Z res = mod(**inputs) 2025-10-10T01:04:48.2374214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2374301Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2374536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2374600Z outputs = layer_module( 2025-10-10T01:04:48.2374842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2374907Z outputs = self.rel_attn( 2025-10-10T01:04:48.2375148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2375218Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2375480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2375611Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2375615Z 2025-10-10T01:04:48.2375717Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2375916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2375982Z res = mod(**inputs) 2025-10-10T01:04:48.2376230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2376320Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2376569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2376642Z outputs = layer_module( 2025-10-10T01:04:48.2376892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2376968Z outputs = self.rel_attn( 2025-10-10T01:04:48.2377216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2377334Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2377338Z 2025-10-10T01:04:48.2377446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2377650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2377720Z res = mod(**inputs) 2025-10-10T01:04:48.2377961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2378040Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2378313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2378382Z outputs = layer_module( 2025-10-10T01:04:48.2378639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2378709Z outputs = self.rel_attn( 2025-10-10T01:04:48.2378980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2379050Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2379309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2379437Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2379441Z 2025-10-10T01:04:48.2379542Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2379766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2379845Z res = mod(**inputs) 2025-10-10T01:04:48.2380095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2380185Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2380438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2380511Z outputs = layer_module( 2025-10-10T01:04:48.2380761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2380829Z outputs = self.rel_attn( 2025-10-10T01:04:48.2381083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2381171Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2381449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2381564Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2381567Z 2025-10-10T01:04:48.2381675Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2381869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2381934Z res = mod(**inputs) 2025-10-10T01:04:48.2382200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2382284Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2382549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2382619Z outputs = layer_module( 2025-10-10T01:04:48.2382875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2382954Z outputs = self.rel_attn( 2025-10-10T01:04:48.2383209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2383325Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2383600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2383721Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2383725Z 2025-10-10T01:04:48.2383830Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2384028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2384103Z res = mod(**inputs) 2025-10-10T01:04:48.2384376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2384471Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2384728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2384800Z outputs = layer_module( 2025-10-10T01:04:48.2385061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2385270Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2385541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2385623Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2385886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2385992Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2386314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2386405Z output = self.layer_1(output) 2025-10-10T01:04:48.2386412Z 2025-10-10T01:04:48.2386521Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2386731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2386801Z res = mod(**inputs) 2025-10-10T01:04:48.2387066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2387174Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2387445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2387529Z outputs = layer_module( 2025-10-10T01:04:48.2387803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2388034Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2388316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2388394Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2388655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2388729Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2388985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2389074Z output = self.activation_function(output) 2025-10-10T01:04:48.2389288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2389365Z return self.act(input) 2025-10-10T01:04:48.2389368Z 2025-10-10T01:04:48.2389471Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2389690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2389756Z res = mod(**inputs) 2025-10-10T01:04:48.2390013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2390096Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2390344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2390420Z outputs = layer_module( 2025-10-10T01:04:48.2390683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2390895Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2391150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2391230Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2391486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2391560Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2391814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2391886Z output = self.layer_2(output) 2025-10-10T01:04:48.2391889Z 2025-10-10T01:04:48.2392001Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2392212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2392320Z res = mod(**inputs) 2025-10-10T01:04:48.2392575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2392658Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2392915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2392984Z outputs = layer_module( 2025-10-10T01:04:48.2393233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2393308Z outputs = self.rel_attn( 2025-10-10T01:04:48.2393557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2393662Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2393667Z 2025-10-10T01:04:48.2393770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2393971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2394034Z res = mod(**inputs) 2025-10-10T01:04:48.2394285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2394372Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2394623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2394696Z outputs = layer_module( 2025-10-10T01:04:48.2394954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2395021Z outputs = self.rel_attn( 2025-10-10T01:04:48.2395275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2395375Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2395379Z 2025-10-10T01:04:48.2395486Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2395688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2395750Z res = mod(**inputs) 2025-10-10T01:04:48.2396000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2396079Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2396329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2396393Z outputs = layer_module( 2025-10-10T01:04:48.2396655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2396726Z outputs = self.rel_attn( 2025-10-10T01:04:48.2396966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2397045Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2397305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2397437Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2397441Z 2025-10-10T01:04:48.2397540Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2397730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2397801Z res = mod(**inputs) 2025-10-10T01:04:48.2398047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2398163Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2398410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2398483Z outputs = layer_module( 2025-10-10T01:04:48.2398728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2398795Z outputs = self.rel_attn( 2025-10-10T01:04:48.2399046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2399173Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2399176Z 2025-10-10T01:04:48.2399283Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2399473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2399541Z res = mod(**inputs) 2025-10-10T01:04:48.2399804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2399886Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2400148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2400222Z outputs = layer_module( 2025-10-10T01:04:48.2400473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2400540Z outputs = self.rel_attn( 2025-10-10T01:04:48.2400784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2400864Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2401127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2401261Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2401264Z 2025-10-10T01:04:48.2401363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2401567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2401638Z res = mod(**inputs) 2025-10-10T01:04:48.2401884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2401970Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2402217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2402283Z outputs = layer_module( 2025-10-10T01:04:48.2402557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2402625Z outputs = self.rel_attn( 2025-10-10T01:04:48.2402874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2402971Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2402976Z 2025-10-10T01:04:48.2403082Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2403271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2403333Z res = mod(**inputs) 2025-10-10T01:04:48.2403586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2403665Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2403925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2404031Z outputs = layer_module( 2025-10-10T01:04:48.2404278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2404353Z outputs = self.rel_attn( 2025-10-10T01:04:48.2404606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2404689Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2404959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2405102Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2405106Z 2025-10-10T01:04:48.2405209Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2405399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2405472Z res = mod(**inputs) 2025-10-10T01:04:48.2405725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2405814Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2406063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2406131Z outputs = layer_module( 2025-10-10T01:04:48.2406394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2406460Z outputs = self.rel_attn( 2025-10-10T01:04:48.2406710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2406798Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2407060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2407176Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2407180Z 2025-10-10T01:04:48.2407277Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2407472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2407557Z res = mod(**inputs) 2025-10-10T01:04:48.2407813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2407894Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2408141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2408215Z outputs = layer_module( 2025-10-10T01:04:48.2408464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2408556Z outputs = self.rel_attn( 2025-10-10T01:04:48.2408805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2408893Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2409167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2409277Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2409280Z 2025-10-10T01:04:48.2409399Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2409587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2409659Z res = mod(**inputs) 2025-10-10T01:04:48.2409902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2410015Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2410268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2410334Z outputs = layer_module( 2025-10-10T01:04:48.2410586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2410783Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2411033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2411116Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2411361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2411440Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2411688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2411766Z output = self.layer_1(output) 2025-10-10T01:04:48.2411770Z 2025-10-10T01:04:48.2411870Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2412058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2412128Z res = mod(**inputs) 2025-10-10T01:04:48.2412372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2412461Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2412704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2412772Z outputs = layer_module( 2025-10-10T01:04:48.2413033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2413237Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2413502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2413597Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2413853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2413925Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2414173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2414272Z output = self.activation_function(output) 2025-10-10T01:04:48.2414500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2414582Z return self.act(input) 2025-10-10T01:04:48.2414585Z 2025-10-10T01:04:48.2414692Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2414888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2414963Z res = mod(**inputs) 2025-10-10T01:04:48.2415217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2415306Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2415562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2415637Z outputs = layer_module( 2025-10-10T01:04:48.2415893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2416112Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2416396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2416473Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2416731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2416802Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2417052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2417131Z output = self.layer_2(output) 2025-10-10T01:04:48.2417134Z 2025-10-10T01:04:48.2417237Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2417439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2417508Z res = mod(**inputs) 2025-10-10T01:04:48.2417768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2417848Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2418098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2418174Z outputs = layer_module( 2025-10-10T01:04:48.2418424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2418500Z outputs = self.rel_attn( 2025-10-10T01:04:48.2418750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2418846Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2418850Z 2025-10-10T01:04:48.2418960Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2419159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2419236Z res = mod(**inputs) 2025-10-10T01:04:48.2419486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2419587Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2419848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2419915Z outputs = layer_module( 2025-10-10T01:04:48.2420169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2420239Z outputs = self.rel_attn( 2025-10-10T01:04:48.2420494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2420612Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2420616Z 2025-10-10T01:04:48.2420719Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2420918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2420983Z res = mod(**inputs) 2025-10-10T01:04:48.2421241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2421322Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2421574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2421648Z outputs = layer_module( 2025-10-10T01:04:48.2421896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2421973Z outputs = self.rel_attn( 2025-10-10T01:04:48.2422252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2422332Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2422608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2422744Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2422747Z 2025-10-10T01:04:48.2422857Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2423059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2423133Z res = mod(**inputs) 2025-10-10T01:04:48.2423396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2423479Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2423750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2423819Z outputs = layer_module( 2025-10-10T01:04:48.2424083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2424155Z outputs = self.rel_attn( 2025-10-10T01:04:48.2424409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2424548Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2424551Z 2025-10-10T01:04:48.2424655Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2424862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2424929Z res = mod(**inputs) 2025-10-10T01:04:48.2425196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2425281Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2425539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2425641Z outputs = layer_module( 2025-10-10T01:04:48.2425898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2425975Z outputs = self.rel_attn( 2025-10-10T01:04:48.2426295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2426381Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2426683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2426846Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2426851Z 2025-10-10T01:04:48.2426975Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2427193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2427275Z res = mod(**inputs) 2025-10-10T01:04:48.2427556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2427648Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2427937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2428015Z outputs = layer_module( 2025-10-10T01:04:48.2428301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2428387Z outputs = self.rel_attn( 2025-10-10T01:04:48.2428676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2428789Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2428793Z 2025-10-10T01:04:48.2428900Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2429112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2429178Z res = mod(**inputs) 2025-10-10T01:04:48.2429436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2429518Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2429767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2429844Z outputs = layer_module( 2025-10-10T01:04:48.2430096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2430170Z outputs = self.rel_attn( 2025-10-10T01:04:48.2430419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2430494Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2430766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2430888Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2430891Z 2025-10-10T01:04:48.2431001Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2431193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2431258Z res = mod(**inputs) 2025-10-10T01:04:48.2431655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2431746Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2432014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2432133Z outputs = layer_module( 2025-10-10T01:04:48.2432399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2432471Z outputs = self.rel_attn( 2025-10-10T01:04:48.2432727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2432828Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2433106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2433270Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2433274Z 2025-10-10T01:04:48.2433379Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2433574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2433648Z res = mod(**inputs) 2025-10-10T01:04:48.2433901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2433990Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2434240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2434316Z outputs = layer_module( 2025-10-10T01:04:48.2434570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2434640Z outputs = self.rel_attn( 2025-10-10T01:04:48.2434946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2435039Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2435323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2435437Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2435441Z 2025-10-10T01:04:48.2435546Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2435752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2435819Z res = mod(**inputs) 2025-10-10T01:04:48.2436082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2436167Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2436438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2436508Z outputs = layer_module( 2025-10-10T01:04:48.2436766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2436984Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2437250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2437337Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2437599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2437674Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2437942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2438020Z output = self.layer_1(output) 2025-10-10T01:04:48.2438023Z 2025-10-10T01:04:48.2438136Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2438339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2438430Z res = mod(**inputs) 2025-10-10T01:04:48.2438744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2438828Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2439089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2439159Z outputs = layer_module( 2025-10-10T01:04:48.2439420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2439651Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2439916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2440006Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2440264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2440343Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2440652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2440751Z output = self.activation_function(output) 2025-10-10T01:04:48.2440971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2441043Z return self.act(input) 2025-10-10T01:04:48.2441079Z 2025-10-10T01:04:48.2441194Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2441395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2441468Z res = mod(**inputs) 2025-10-10T01:04:48.2441726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2441811Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2442076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2442147Z outputs = layer_module( 2025-10-10T01:04:48.2442410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2442619Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2442892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2442973Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2443234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2443320Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2443579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2443661Z output = self.layer_2(output) 2025-10-10T01:04:48.2443664Z 2025-10-10T01:04:48.2443770Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2443974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2444048Z res = mod(**inputs) 2025-10-10T01:04:48.2444312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2444404Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2444664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2444754Z outputs = layer_module( 2025-10-10T01:04:48.2445013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2445085Z outputs = self.rel_attn( 2025-10-10T01:04:48.2445346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2445447Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2445451Z 2025-10-10T01:04:48.2445564Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2445782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2445860Z res = mod(**inputs) 2025-10-10T01:04:48.2446114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2446193Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2446445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2446512Z outputs = layer_module( 2025-10-10T01:04:48.2446762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2446840Z outputs = self.rel_attn( 2025-10-10T01:04:48.2447104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2447213Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2447264Z 2025-10-10T01:04:48.2447368Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2447569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2447634Z res = mod(**inputs) 2025-10-10T01:04:48.2447891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2447981Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2448237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2448313Z outputs = layer_module( 2025-10-10T01:04:48.2448574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2448640Z outputs = self.rel_attn( 2025-10-10T01:04:48.2448893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2448967Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2449234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2449363Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2449366Z 2025-10-10T01:04:48.2449472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2449661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2449725Z res = mod(**inputs) 2025-10-10T01:04:48.2449980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2450060Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2450315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2450383Z outputs = layer_module( 2025-10-10T01:04:48.2450625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2450718Z outputs = self.rel_attn( 2025-10-10T01:04:48.2450965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2451098Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2451101Z 2025-10-10T01:04:48.2451201Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2451391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2451461Z res = mod(**inputs) 2025-10-10T01:04:48.2451722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2451811Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2452052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2452126Z outputs = layer_module( 2025-10-10T01:04:48.2452365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2452431Z outputs = self.rel_attn( 2025-10-10T01:04:48.2452679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2452750Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2453012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2453137Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2453168Z 2025-10-10T01:04:48.2453270Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2453466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2453529Z res = mod(**inputs) 2025-10-10T01:04:48.2453778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2453858Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2454105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2454171Z outputs = layer_module( 2025-10-10T01:04:48.2454411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2454486Z outputs = self.rel_attn( 2025-10-10T01:04:48.2454737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2454843Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2454847Z 2025-10-10T01:04:48.2454948Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2455151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2455221Z res = mod(**inputs) 2025-10-10T01:04:48.2455464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2455549Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2455789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2455855Z outputs = layer_module( 2025-10-10T01:04:48.2456103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2456172Z outputs = self.rel_attn( 2025-10-10T01:04:48.2456416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2456506Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2456779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2456904Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2456907Z 2025-10-10T01:04:48.2457012Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2457217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2457282Z res = mod(**inputs) 2025-10-10T01:04:48.2457547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2457649Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2457900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2457974Z outputs = layer_module( 2025-10-10T01:04:48.2458224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2458300Z outputs = self.rel_attn( 2025-10-10T01:04:48.2458548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2458642Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2458908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2459020Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2459040Z 2025-10-10T01:04:48.2459167Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2459368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2459440Z res = mod(**inputs) 2025-10-10T01:04:48.2459685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2459766Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2460024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2460092Z outputs = layer_module( 2025-10-10T01:04:48.2460345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2460414Z outputs = self.rel_attn( 2025-10-10T01:04:48.2460665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2460764Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2461031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2461156Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2461159Z 2025-10-10T01:04:48.2461264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2461469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2461535Z res = mod(**inputs) 2025-10-10T01:04:48.2461795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2461887Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2462146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2462225Z outputs = layer_module( 2025-10-10T01:04:48.2462479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2462703Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2462977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2463060Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2463323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2463401Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2463679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2463776Z output = self.layer_1(output) 2025-10-10T01:04:48.2463780Z 2025-10-10T01:04:48.2463891Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2464114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2464185Z res = mod(**inputs) 2025-10-10T01:04:48.2464466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2464555Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2464828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2464909Z outputs = layer_module( 2025-10-10T01:04:48.2465178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2465424Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2465723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2465817Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2466092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2466340Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2466641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2466736Z output = self.activation_function(output) 2025-10-10T01:04:48.2466983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2467060Z return self.act(input) 2025-10-10T01:04:48.2467064Z 2025-10-10T01:04:48.2467183Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2467405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2467476Z res = mod(**inputs) 2025-10-10T01:04:48.2467762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2467847Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2468119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2468189Z outputs = layer_module( 2025-10-10T01:04:48.2468451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2468664Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2468933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2469023Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2469287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2469384Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2469645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2469718Z output = self.layer_2(output) 2025-10-10T01:04:48.2469721Z 2025-10-10T01:04:48.2469836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2470035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2470107Z res = mod(**inputs) 2025-10-10T01:04:48.2470382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2470469Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2470732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2470803Z outputs = layer_module( 2025-10-10T01:04:48.2471063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2471133Z outputs = self.rel_attn( 2025-10-10T01:04:48.2471387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2471496Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2471500Z 2025-10-10T01:04:48.2471603Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2471810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2471922Z res = mod(**inputs) 2025-10-10T01:04:48.2472185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2472269Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2472525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2472605Z outputs = layer_module( 2025-10-10T01:04:48.2472861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2472938Z outputs = self.rel_attn( 2025-10-10T01:04:48.2473190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2473291Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2473295Z 2025-10-10T01:04:48.2473409Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2473608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2473683Z res = mod(**inputs) 2025-10-10T01:04:48.2473939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2474025Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2474288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2474357Z outputs = layer_module( 2025-10-10T01:04:48.2474625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2474693Z outputs = self.rel_attn( 2025-10-10T01:04:48.2474938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2475013Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2475273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2475407Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2475427Z 2025-10-10T01:04:48.2475529Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2475721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2475785Z res = mod(**inputs) 2025-10-10T01:04:48.2476026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2476114Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2476355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2476448Z outputs = layer_module( 2025-10-10T01:04:48.2476691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2476764Z outputs = self.rel_attn( 2025-10-10T01:04:48.2477008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2477135Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2477138Z 2025-10-10T01:04:48.2477247Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2477436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2477506Z res = mod(**inputs) 2025-10-10T01:04:48.2477756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2477874Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2478138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2478205Z outputs = layer_module( 2025-10-10T01:04:48.2478460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2478530Z outputs = self.rel_attn( 2025-10-10T01:04:48.2478779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2478859Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2479125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2479259Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2479262Z 2025-10-10T01:04:48.2479376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2479572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2479636Z res = mod(**inputs) 2025-10-10T01:04:48.2479881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2479971Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2480215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2480287Z outputs = layer_module( 2025-10-10T01:04:48.2480529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2480596Z outputs = self.rel_attn( 2025-10-10T01:04:48.2480849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2480948Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2480952Z 2025-10-10T01:04:48.2481061Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2481252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2481342Z res = mod(**inputs) 2025-10-10T01:04:48.2481587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2481667Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2481923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2481989Z outputs = layer_module( 2025-10-10T01:04:48.2482245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2482329Z outputs = self.rel_attn( 2025-10-10T01:04:48.2482582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2482658Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2482911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2483038Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2483042Z 2025-10-10T01:04:48.2483140Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2483335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2483398Z res = mod(**inputs) 2025-10-10T01:04:48.2483643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2483729Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2484011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2484083Z outputs = layer_module( 2025-10-10T01:04:48.2484326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2484394Z outputs = self.rel_attn( 2025-10-10T01:04:48.2484644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2484731Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2485007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2485116Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2485120Z 2025-10-10T01:04:48.2485226Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2485423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2485486Z res = mod(**inputs) 2025-10-10T01:04:48.2485744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2485825Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2486079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2486146Z outputs = layer_module( 2025-10-10T01:04:48.2486393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2486467Z outputs = self.rel_attn( 2025-10-10T01:04:48.2486713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2486812Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2487078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2487188Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2487214Z 2025-10-10T01:04:48.2487317Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2487510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2487584Z res = mod(**inputs) 2025-10-10T01:04:48.2487844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2487931Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2488177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2488260Z outputs = layer_module( 2025-10-10T01:04:48.2488510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2488707Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2488963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2489041Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2489283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2489363Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2489604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2489682Z output = self.layer_1(output) 2025-10-10T01:04:48.2489726Z 2025-10-10T01:04:48.2489845Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2490045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2490108Z res = mod(**inputs) 2025-10-10T01:04:48.2490352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2490440Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2490683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2490757Z outputs = layer_module( 2025-10-10T01:04:48.2491003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2491203Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2491469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2491546Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2491801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2491875Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2492129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2492217Z output = self.activation_function(output) 2025-10-10T01:04:48.2492426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2492506Z return self.act(input) 2025-10-10T01:04:48.2492509Z 2025-10-10T01:04:48.2492611Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2492816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2492881Z res = mod(**inputs) 2025-10-10T01:04:48.2493131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2493240Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2493496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2493573Z outputs = layer_module( 2025-10-10T01:04:48.2493828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2494041Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2494330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2494418Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2494679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2494753Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2495017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2495101Z output = self.layer_2(output) 2025-10-10T01:04:48.2495105Z 2025-10-10T01:04:48.2495207Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2495408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2495475Z res = mod(**inputs) 2025-10-10T01:04:48.2495733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2495828Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2496094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2496171Z outputs = layer_module( 2025-10-10T01:04:48.2496422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2496502Z outputs = self.rel_attn( 2025-10-10T01:04:48.2496756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2496863Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2496867Z 2025-10-10T01:04:48.2496983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2497175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2497247Z res = mod(**inputs) 2025-10-10T01:04:48.2497501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2497589Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2497839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2497907Z outputs = layer_module( 2025-10-10T01:04:48.2498166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2498236Z outputs = self.rel_attn( 2025-10-10T01:04:48.2498499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2498599Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2498603Z 2025-10-10T01:04:48.2498713Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2498923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2498987Z res = mod(**inputs) 2025-10-10T01:04:48.2499246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2499343Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2499606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2499675Z outputs = layer_module( 2025-10-10T01:04:48.2499930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2500007Z outputs = self.rel_attn( 2025-10-10T01:04:48.2500263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2500348Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2500638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2500774Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2500786Z 2025-10-10T01:04:48.2500890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2501089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2501163Z res = mod(**inputs) 2025-10-10T01:04:48.2501423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2501512Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2501768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2501837Z outputs = layer_module( 2025-10-10T01:04:48.2502136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2502209Z outputs = self.rel_attn( 2025-10-10T01:04:48.2502504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2502650Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2502654Z 2025-10-10T01:04:48.2502773Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2502983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2503055Z res = mod(**inputs) 2025-10-10T01:04:48.2503336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2503425Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2503707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2503783Z outputs = layer_module( 2025-10-10T01:04:48.2504055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2504140Z outputs = self.rel_attn( 2025-10-10T01:04:48.2504412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2504500Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2504788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2504926Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2504939Z 2025-10-10T01:04:48.2505050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2505267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2505345Z res = mod(**inputs) 2025-10-10T01:04:48.2505662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2505775Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2506047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2506119Z outputs = layer_module( 2025-10-10T01:04:48.2506482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2506563Z outputs = self.rel_attn( 2025-10-10T01:04:48.2506847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2506977Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2506984Z 2025-10-10T01:04:48.2507097Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2507322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2507396Z res = mod(**inputs) 2025-10-10T01:04:48.2507682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2507772Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2508055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2508128Z outputs = layer_module( 2025-10-10T01:04:48.2508400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2508482Z outputs = self.rel_attn( 2025-10-10T01:04:48.2508772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2508877Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2509173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2509308Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2509312Z 2025-10-10T01:04:48.2509431Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2509644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2509721Z res = mod(**inputs) 2025-10-10T01:04:48.2510007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2510094Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2510386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2510460Z outputs = layer_module( 2025-10-10T01:04:48.2510747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2510823Z outputs = self.rel_attn( 2025-10-10T01:04:48.2511107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2511204Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2511506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2511638Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2511642Z 2025-10-10T01:04:48.2511753Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2511980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2512053Z res = mod(**inputs) 2025-10-10T01:04:48.2512357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2512470Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2512737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2512816Z outputs = layer_module( 2025-10-10T01:04:48.2513084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2513165Z outputs = self.rel_attn( 2025-10-10T01:04:48.2513431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2513526Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2513841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2513964Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2513967Z 2025-10-10T01:04:48.2514086Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2514295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2514367Z res = mod(**inputs) 2025-10-10T01:04:48.2514644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2514730Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2532712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2532895Z outputs = layer_module( 2025-10-10T01:04:48.2533630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2533858Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2534144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2534232Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2534512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2534593Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2534864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2534944Z output = self.layer_1(output) 2025-10-10T01:04:48.2534949Z 2025-10-10T01:04:48.2535069Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2535297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2535369Z res = mod(**inputs) 2025-10-10T01:04:48.2535635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2535729Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2535995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2536070Z outputs = layer_module( 2025-10-10T01:04:48.2536326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2536545Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2536813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2536909Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2537166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2537279Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2537540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2537633Z output = self.activation_function(output) 2025-10-10T01:04:48.2537859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2537933Z return self.act(input) 2025-10-10T01:04:48.2537937Z 2025-10-10T01:04:48.2538058Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2538265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2538373Z res = mod(**inputs) 2025-10-10T01:04:48.2538643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2538734Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2538994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2539067Z outputs = layer_module( 2025-10-10T01:04:48.2539315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2539528Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2539787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2539873Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2540156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2540239Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2540492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2540568Z output = self.layer_2(output) 2025-10-10T01:04:48.2540572Z 2025-10-10T01:04:48.2540687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2540891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2540966Z res = mod(**inputs) 2025-10-10T01:04:48.2541220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2541308Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2541573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2541645Z outputs = layer_module( 2025-10-10T01:04:48.2541903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2541977Z outputs = self.rel_attn( 2025-10-10T01:04:48.2542226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2542337Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2542340Z 2025-10-10T01:04:48.2542446Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2542655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2542721Z res = mod(**inputs) 2025-10-10T01:04:48.2542990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2543080Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2543338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2543434Z outputs = layer_module( 2025-10-10T01:04:48.2543690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2543770Z outputs = self.rel_attn( 2025-10-10T01:04:48.2544028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2544136Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2544148Z 2025-10-10T01:04:48.2544256Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2544459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2544556Z res = mod(**inputs) 2025-10-10T01:04:48.2544823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2544917Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2545178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2545250Z outputs = layer_module( 2025-10-10T01:04:48.2545519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2545591Z outputs = self.rel_attn( 2025-10-10T01:04:48.2545854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2545936Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2546304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2546480Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2546485Z 2025-10-10T01:04:48.2546591Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2546806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2546874Z res = mod(**inputs) 2025-10-10T01:04:48.2547148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2547233Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2547495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2547576Z outputs = layer_module( 2025-10-10T01:04:48.2547844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2547924Z outputs = self.rel_attn( 2025-10-10T01:04:48.2548176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2548320Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2548324Z 2025-10-10T01:04:48.2548438Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2548634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2548708Z res = mod(**inputs) 2025-10-10T01:04:48.2548968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2549052Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2549321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2549391Z outputs = layer_module( 2025-10-10T01:04:48.2549649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2549719Z outputs = self.rel_attn( 2025-10-10T01:04:48.2549995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2550071Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2550340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2550479Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2550482Z 2025-10-10T01:04:48.2550585Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2550788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2550881Z res = mod(**inputs) 2025-10-10T01:04:48.2551137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2551229Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2551482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2551558Z outputs = layer_module( 2025-10-10T01:04:48.2551809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2551887Z outputs = self.rel_attn( 2025-10-10T01:04:48.2552137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2552263Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2552266Z 2025-10-10T01:04:48.2552414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2552602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2552673Z res = mod(**inputs) 2025-10-10T01:04:48.2552915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2552996Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2553244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2553310Z outputs = layer_module( 2025-10-10T01:04:48.2553556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2553622Z outputs = self.rel_attn( 2025-10-10T01:04:48.2553862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2553941Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2554196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2554325Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2554329Z 2025-10-10T01:04:48.2554430Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2554627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2554692Z res = mod(**inputs) 2025-10-10T01:04:48.2554961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2555052Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2555306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2555384Z outputs = layer_module( 2025-10-10T01:04:48.2555637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2555706Z outputs = self.rel_attn( 2025-10-10T01:04:48.2555985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2556078Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2556366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2556479Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2556483Z 2025-10-10T01:04:48.2556587Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2556777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2556869Z res = mod(**inputs) 2025-10-10T01:04:48.2557120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2557198Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2557447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2557512Z outputs = layer_module( 2025-10-10T01:04:48.2557760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2557835Z outputs = self.rel_attn( 2025-10-10T01:04:48.2558084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2558177Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2558473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2558606Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2558610Z 2025-10-10T01:04:48.2558711Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2558903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2558974Z res = mod(**inputs) 2025-10-10T01:04:48.2559274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2559363Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2559602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2559666Z outputs = layer_module( 2025-10-10T01:04:48.2559913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2560111Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2560367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2560447Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2560700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2560775Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2561021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2561105Z output = self.layer_1(output) 2025-10-10T01:04:48.2561108Z 2025-10-10T01:04:48.2561210Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2561410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2561477Z res = mod(**inputs) 2025-10-10T01:04:48.2561720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2561823Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2562071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2562147Z outputs = layer_module( 2025-10-10T01:04:48.2562401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2562602Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2562848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2562941Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2563192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2563262Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2563513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2563600Z output = self.activation_function(output) 2025-10-10T01:04:48.2563812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2563889Z return self.act(input) 2025-10-10T01:04:48.2563893Z 2025-10-10T01:04:48.2563994Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2564194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2564258Z res = mod(**inputs) 2025-10-10T01:04:48.2564547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2564639Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2564891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2564968Z outputs = layer_module( 2025-10-10T01:04:48.2565217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2565427Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2565684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2565763Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2566026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2566100Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2566357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2566433Z output = self.layer_2(output) 2025-10-10T01:04:48.2566437Z 2025-10-10T01:04:48.2566548Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2566746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2566812Z res = mod(**inputs) 2025-10-10T01:04:48.2567074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2567156Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2567428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2567497Z outputs = layer_module( 2025-10-10T01:04:48.2567740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2567818Z outputs = self.rel_attn( 2025-10-10T01:04:48.2568077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2568183Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2568187Z 2025-10-10T01:04:48.2568287Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2568480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2568556Z res = mod(**inputs) 2025-10-10T01:04:48.2568808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2568918Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2569169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2569247Z outputs = layer_module( 2025-10-10T01:04:48.2569497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2569568Z outputs = self.rel_attn( 2025-10-10T01:04:48.2569824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2569926Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2569930Z 2025-10-10T01:04:48.2570043Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2570239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2570304Z res = mod(**inputs) 2025-10-10T01:04:48.2570598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2570681Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2570933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2571002Z outputs = layer_module( 2025-10-10T01:04:48.2571247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2571325Z outputs = self.rel_attn( 2025-10-10T01:04:48.2571572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2571652Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2571917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2572064Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2572068Z 2025-10-10T01:04:48.2572170Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2572362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2572439Z res = mod(**inputs) 2025-10-10T01:04:48.2572688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2572778Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2573026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2573095Z outputs = layer_module( 2025-10-10T01:04:48.2573355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2573428Z outputs = self.rel_attn( 2025-10-10T01:04:48.2573682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2573816Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2573834Z 2025-10-10T01:04:48.2573943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2574137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2574204Z res = mod(**inputs) 2025-10-10T01:04:48.2574463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2574545Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2574802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2574873Z outputs = layer_module( 2025-10-10T01:04:48.2575153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2575231Z outputs = self.rel_attn( 2025-10-10T01:04:48.2575485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2575571Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2575849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2575993Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2575997Z 2025-10-10T01:04:48.2576110Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2576309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2576383Z res = mod(**inputs) 2025-10-10T01:04:48.2576665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2576756Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2577010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2577079Z outputs = layer_module( 2025-10-10T01:04:48.2577337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2577406Z outputs = self.rel_attn( 2025-10-10T01:04:48.2577668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2577770Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2577774Z 2025-10-10T01:04:48.2577885Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2578084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2578150Z res = mod(**inputs) 2025-10-10T01:04:48.2578411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2578497Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2578756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2578823Z outputs = layer_module( 2025-10-10T01:04:48.2579074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2579150Z outputs = self.rel_attn( 2025-10-10T01:04:48.2579402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2579484Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2579756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2579885Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2579913Z 2025-10-10T01:04:48.2580019Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2580217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2580293Z res = mod(**inputs) 2025-10-10T01:04:48.2580554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2580647Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2580906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2580975Z outputs = layer_module( 2025-10-10T01:04:48.2581260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2581335Z outputs = self.rel_attn( 2025-10-10T01:04:48.2581603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2581702Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2581989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2582119Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2582123Z 2025-10-10T01:04:48.2582232Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2582458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2582533Z res = mod(**inputs) 2025-10-10T01:04:48.2582850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2582940Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2583214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2583302Z outputs = layer_module( 2025-10-10T01:04:48.2583572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2583659Z outputs = self.rel_attn( 2025-10-10T01:04:48.2583931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2584028Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2584333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2584461Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2584465Z 2025-10-10T01:04:48.2584587Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2584806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2584884Z res = mod(**inputs) 2025-10-10T01:04:48.2585148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2585233Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2585512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2585586Z outputs = layer_module( 2025-10-10T01:04:48.2585864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2586093Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2586478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2586609Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2586891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2586981Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2587260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2587342Z output = self.layer_1(output) 2025-10-10T01:04:48.2587354Z 2025-10-10T01:04:48.2587467Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2587692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2587791Z res = mod(**inputs) 2025-10-10T01:04:48.2588062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2588160Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2588436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2588510Z outputs = layer_module( 2025-10-10T01:04:48.2588791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2589010Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2589298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2589383Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2589704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2589786Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2590064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2590169Z output = self.activation_function(output) 2025-10-10T01:04:48.2590403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2590490Z return self.act(input) 2025-10-10T01:04:48.2590495Z 2025-10-10T01:04:48.2590607Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2590821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2590902Z res = mod(**inputs) 2025-10-10T01:04:48.2591164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2591253Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2591502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2591570Z outputs = layer_module( 2025-10-10T01:04:48.2591823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2592021Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2592279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2592355Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2592609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2592692Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2592935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2593016Z output = self.layer_2(output) 2025-10-10T01:04:48.2593035Z 2025-10-10T01:04:48.2593138Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2593327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2593398Z res = mod(**inputs) 2025-10-10T01:04:48.2593643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2593727Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2593969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2594047Z outputs = layer_module( 2025-10-10T01:04:48.2594313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2594383Z outputs = self.rel_attn( 2025-10-10T01:04:48.2594622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2594724Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2594728Z 2025-10-10T01:04:48.2594828Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2595022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2595085Z res = mod(**inputs) 2025-10-10T01:04:48.2595335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2595414Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2595690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2595763Z outputs = layer_module( 2025-10-10T01:04:48.2596004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2596079Z outputs = self.rel_attn( 2025-10-10T01:04:48.2596318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2596414Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2596417Z 2025-10-10T01:04:48.2596523Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2596707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2596777Z res = mod(**inputs) 2025-10-10T01:04:48.2597021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2597101Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2597348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2597416Z outputs = layer_module( 2025-10-10T01:04:48.2597663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2597732Z outputs = self.rel_attn( 2025-10-10T01:04:48.2597979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2598050Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2598306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2598440Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2598445Z 2025-10-10T01:04:48.2598545Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2598740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2598855Z res = mod(**inputs) 2025-10-10T01:04:48.2599101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2599188Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2599430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2599501Z outputs = layer_module( 2025-10-10T01:04:48.2599744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2599816Z outputs = self.rel_attn( 2025-10-10T01:04:48.2600076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2600206Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2600210Z 2025-10-10T01:04:48.2600315Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2600504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2600572Z res = mod(**inputs) 2025-10-10T01:04:48.2600817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2600896Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2601145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2601210Z outputs = layer_module( 2025-10-10T01:04:48.2601474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2601554Z outputs = self.rel_attn( 2025-10-10T01:04:48.2601799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2601870Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2602127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2602257Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2602260Z 2025-10-10T01:04:48.2602358Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2602552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2602615Z res = mod(**inputs) 2025-10-10T01:04:48.2602859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2602945Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2603186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2603260Z outputs = layer_module( 2025-10-10T01:04:48.2603504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2603572Z outputs = self.rel_attn( 2025-10-10T01:04:48.2603826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2603924Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2603927Z 2025-10-10T01:04:48.2604034Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2604231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2604305Z res = mod(**inputs) 2025-10-10T01:04:48.2604557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2604639Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2604914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2604983Z outputs = layer_module( 2025-10-10T01:04:48.2605253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2605320Z outputs = self.rel_attn( 2025-10-10T01:04:48.2605562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2605639Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2605911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2606041Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2606045Z 2025-10-10T01:04:48.2606145Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2606345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2606411Z res = mod(**inputs) 2025-10-10T01:04:48.2606653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2606740Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2606985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2607059Z outputs = layer_module( 2025-10-10T01:04:48.2607316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2607398Z outputs = self.rel_attn( 2025-10-10T01:04:48.2607653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2607744Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2608021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2608135Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2608138Z 2025-10-10T01:04:48.2608244Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2608437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2608502Z res = mod(**inputs) 2025-10-10T01:04:48.2608763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2608846Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2609102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2609171Z outputs = layer_module( 2025-10-10T01:04:48.2609424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2609501Z outputs = self.rel_attn( 2025-10-10T01:04:48.2609749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2609842Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2610110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2610221Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2610234Z 2025-10-10T01:04:48.2610336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2610530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2610602Z res = mod(**inputs) 2025-10-10T01:04:48.2610868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2610956Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2611205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2611272Z outputs = layer_module( 2025-10-10T01:04:48.2611525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2611748Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2612014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2612092Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2612340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2612421Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2612667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2612748Z output = self.layer_1(output) 2025-10-10T01:04:48.2612751Z 2025-10-10T01:04:48.2612853Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2613053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2613118Z res = mod(**inputs) 2025-10-10T01:04:48.2613397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2613484Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2613735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2613811Z outputs = layer_module( 2025-10-10T01:04:48.2614062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2614263Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2614527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2614603Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2614861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2614934Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2615191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2615281Z output = self.activation_function(output) 2025-10-10T01:04:48.2615495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2615571Z return self.act(input) 2025-10-10T01:04:48.2615574Z 2025-10-10T01:04:48.2615675Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2615877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2615940Z res = mod(**inputs) 2025-10-10T01:04:48.2616193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2616287Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2616537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2616610Z outputs = layer_module( 2025-10-10T01:04:48.2616871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2617078Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2617329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2617406Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2617660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2617732Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2618009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2618087Z output = self.layer_2(output) 2025-10-10T01:04:48.2618091Z 2025-10-10T01:04:48.2618197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2618407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2618485Z res = mod(**inputs) 2025-10-10T01:04:48.2618742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2618823Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2619078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2619146Z outputs = layer_module( 2025-10-10T01:04:48.2619410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2619512Z outputs = self.rel_attn( 2025-10-10T01:04:48.2619765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2619869Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2619872Z 2025-10-10T01:04:48.2619973Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2620167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2620239Z res = mod(**inputs) 2025-10-10T01:04:48.2620492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2620581Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2620832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2620902Z outputs = layer_module( 2025-10-10T01:04:48.2621157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2621228Z outputs = self.rel_attn( 2025-10-10T01:04:48.2621482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2621581Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2621585Z 2025-10-10T01:04:48.2621693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2621887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2621953Z res = mod(**inputs) 2025-10-10T01:04:48.2622211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2622298Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2622554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2622620Z outputs = layer_module( 2025-10-10T01:04:48.2622892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2622970Z outputs = self.rel_attn( 2025-10-10T01:04:48.2623226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2623311Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2623603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2623750Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2623756Z 2025-10-10T01:04:48.2623882Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2624096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2624176Z res = mod(**inputs) 2025-10-10T01:04:48.2624449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2624545Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2624817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2624891Z outputs = layer_module( 2025-10-10T01:04:48.2625166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2625239Z outputs = self.rel_attn( 2025-10-10T01:04:48.2625537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2625698Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2625702Z 2025-10-10T01:04:48.2625817Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2626030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2626099Z res = mod(**inputs) 2025-10-10T01:04:48.2626470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2626567Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2626856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2626932Z outputs = layer_module( 2025-10-10T01:04:48.2627211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2627304Z outputs = self.rel_attn( 2025-10-10T01:04:48.2627583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2627672Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2627978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2628117Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2628129Z 2025-10-10T01:04:48.2628240Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2628449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2628530Z res = mod(**inputs) 2025-10-10T01:04:48.2628802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2628901Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2629174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2629246Z outputs = layer_module( 2025-10-10T01:04:48.2629547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2629622Z outputs = self.rel_attn( 2025-10-10T01:04:48.2629898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2630005Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2630009Z 2025-10-10T01:04:48.2630116Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2630336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2630406Z res = mod(**inputs) 2025-10-10T01:04:48.2630707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2630797Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2631073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2631148Z outputs = layer_module( 2025-10-10T01:04:48.2631417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2631672Z outputs = self.rel_attn( 2025-10-10T01:04:48.2631948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2632032Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2632366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2632522Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2632526Z 2025-10-10T01:04:48.2632645Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2632853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2632932Z res = mod(**inputs) 2025-10-10T01:04:48.2633203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2633290Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2633567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2633641Z outputs = layer_module( 2025-10-10T01:04:48.2633916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2633994Z outputs = self.rel_attn( 2025-10-10T01:04:48.2634269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2634365Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2634657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2634786Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2634790Z 2025-10-10T01:04:48.2634899Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2635118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2635189Z res = mod(**inputs) 2025-10-10T01:04:48.2635461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2635562Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2635832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2635915Z outputs = layer_module( 2025-10-10T01:04:48.2636183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2636288Z outputs = self.rel_attn( 2025-10-10T01:04:48.2636557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2636652Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2636951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2637062Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2637065Z 2025-10-10T01:04:48.2637197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2637394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2637459Z res = mod(**inputs) 2025-10-10T01:04:48.2637722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2637806Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2638074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2638143Z outputs = layer_module( 2025-10-10T01:04:48.2638412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2638613Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2638887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2638992Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2639241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2639322Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2639578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2639650Z output = self.layer_1(output) 2025-10-10T01:04:48.2639653Z 2025-10-10T01:04:48.2639760Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2639949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2640019Z res = mod(**inputs) 2025-10-10T01:04:48.2640265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2640352Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2640595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2640662Z outputs = layer_module( 2025-10-10T01:04:48.2640910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2641108Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2641367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2641447Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2641699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2641783Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2642034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2642129Z output = self.activation_function(output) 2025-10-10T01:04:48.2642356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2642431Z return self.act(input) 2025-10-10T01:04:48.2642435Z 2025-10-10T01:04:48.2642536Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2642728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2642801Z res = mod(**inputs) 2025-10-10T01:04:48.2643052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2643145Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2643414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2643483Z outputs = layer_module( 2025-10-10T01:04:48.2643741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2643939Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2644201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2644278Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2644541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2644615Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2644896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2644997Z output = self.layer_2(output) 2025-10-10T01:04:48.2645001Z 2025-10-10T01:04:48.2645106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2645314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2645383Z res = mod(**inputs) 2025-10-10T01:04:48.2645648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2645735Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2645986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2646069Z outputs = layer_module( 2025-10-10T01:04:48.2646311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2646382Z outputs = self.rel_attn( 2025-10-10T01:04:48.2646631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2646726Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2646730Z 2025-10-10T01:04:48.2646837Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2647026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2647094Z res = mod(**inputs) 2025-10-10T01:04:48.2647339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2647419Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2647676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2647749Z outputs = layer_module( 2025-10-10T01:04:48.2648008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2648080Z outputs = self.rel_attn( 2025-10-10T01:04:48.2648334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2648460Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2648463Z 2025-10-10T01:04:48.2648567Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2648770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2648835Z res = mod(**inputs) 2025-10-10T01:04:48.2649095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2649186Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2649469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2649547Z outputs = layer_module( 2025-10-10T01:04:48.2649798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2649877Z outputs = self.rel_attn( 2025-10-10T01:04:48.2650130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2650205Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2650482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2650612Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2650616Z 2025-10-10T01:04:48.2650728Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2650962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2651033Z res = mod(**inputs) 2025-10-10T01:04:48.2651315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2651406Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2651692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2651762Z outputs = layer_module( 2025-10-10T01:04:48.2652024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2652095Z outputs = self.rel_attn( 2025-10-10T01:04:48.2652348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2652495Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2652498Z 2025-10-10T01:04:48.2652602Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2652805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2652873Z res = mod(**inputs) 2025-10-10T01:04:48.2653134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2653223Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2653472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2653548Z outputs = layer_module( 2025-10-10T01:04:48.2653804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2653888Z outputs = self.rel_attn( 2025-10-10T01:04:48.2654146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2654220Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2654502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2654654Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2654657Z 2025-10-10T01:04:48.2654767Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2654966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2655032Z res = mod(**inputs) 2025-10-10T01:04:48.2655296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2655379Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2655661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2655733Z outputs = layer_module( 2025-10-10T01:04:48.2655994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2656066Z outputs = self.rel_attn( 2025-10-10T01:04:48.2656319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2656427Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2656431Z 2025-10-10T01:04:48.2656533Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2656739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2656805Z res = mod(**inputs) 2025-10-10T01:04:48.2657080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2657188Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2657442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2657522Z outputs = layer_module( 2025-10-10T01:04:48.2657776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2657846Z outputs = self.rel_attn( 2025-10-10T01:04:48.2658105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2658180Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2658454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2658582Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2658587Z 2025-10-10T01:04:48.2658697Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2658895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2658962Z res = mod(**inputs) 2025-10-10T01:04:48.2659225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2659307Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2659567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2659635Z outputs = layer_module( 2025-10-10T01:04:48.2659887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2659963Z outputs = self.rel_attn( 2025-10-10T01:04:48.2660219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2660315Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2660587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2660725Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2660729Z 2025-10-10T01:04:48.2661057Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2661258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2661331Z res = mod(**inputs) 2025-10-10T01:04:48.2661590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2661680Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2661956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2662033Z outputs = layer_module( 2025-10-10T01:04:48.2662310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2662385Z outputs = self.rel_attn( 2025-10-10T01:04:48.2662662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2662757Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2663060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2663179Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2663183Z 2025-10-10T01:04:48.2663292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2663548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2663623Z res = mod(**inputs) 2025-10-10T01:04:48.2663903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2663996Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2664269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2664350Z outputs = layer_module( 2025-10-10T01:04:48.2664619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2664847Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2665130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2665224Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2665499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2665582Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2665858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2665938Z output = self.layer_1(output) 2025-10-10T01:04:48.2665942Z 2025-10-10T01:04:48.2666058Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2666343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2666421Z res = mod(**inputs) 2025-10-10T01:04:48.2666711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2666808Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2667103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2667178Z outputs = layer_module( 2025-10-10T01:04:48.2667465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2667717Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2668007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2668106Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2668397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2668480Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2668766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2668858Z output = self.activation_function(output) 2025-10-10T01:04:48.2669084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2669160Z return self.act(input) 2025-10-10T01:04:48.2669164Z 2025-10-10T01:04:48.2669276Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2669478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2669554Z res = mod(**inputs) 2025-10-10T01:04:48.2669816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2669903Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2670188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2670273Z outputs = layer_module( 2025-10-10T01:04:48.2670532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2670741Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2671004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2671090Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2671346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2671428Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2671683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2671770Z output = self.layer_2(output) 2025-10-10T01:04:48.2671774Z 2025-10-10T01:04:48.2671878Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2672078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2672154Z res = mod(**inputs) 2025-10-10T01:04:48.2672421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2672510Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2672762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2672829Z outputs = layer_module( 2025-10-10T01:04:48.2673084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2673152Z outputs = self.rel_attn( 2025-10-10T01:04:48.2673412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2673509Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2673513Z 2025-10-10T01:04:48.2673637Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2673829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2673894Z res = mod(**inputs) 2025-10-10T01:04:48.2674151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2674232Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2674492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2674559Z outputs = layer_module( 2025-10-10T01:04:48.2674825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2674905Z outputs = self.rel_attn( 2025-10-10T01:04:48.2675157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2675266Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2675270Z 2025-10-10T01:04:48.2675369Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2675563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2675635Z res = mod(**inputs) 2025-10-10T01:04:48.2675885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2675973Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2676239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2676327Z outputs = layer_module( 2025-10-10T01:04:48.2676577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2676646Z outputs = self.rel_attn( 2025-10-10T01:04:48.2676900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2676972Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2677244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2677375Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2677378Z 2025-10-10T01:04:48.2677480Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2677686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2677753Z res = mod(**inputs) 2025-10-10T01:04:48.2678020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2678103Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2678365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2678446Z outputs = layer_module( 2025-10-10T01:04:48.2678695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2678770Z outputs = self.rel_attn( 2025-10-10T01:04:48.2679019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2679157Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2679164Z 2025-10-10T01:04:48.2679264Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2679459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2679533Z res = mod(**inputs) 2025-10-10T01:04:48.2679797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2679885Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2680132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2680200Z outputs = layer_module( 2025-10-10T01:04:48.2680455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2680525Z outputs = self.rel_attn( 2025-10-10T01:04:48.2680794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2680871Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2681145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2681272Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2681275Z 2025-10-10T01:04:48.2681376Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2681577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2681643Z res = mod(**inputs) 2025-10-10T01:04:48.2681907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2681989Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2682256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2682348Z outputs = layer_module( 2025-10-10T01:04:48.2682595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2682674Z outputs = self.rel_attn( 2025-10-10T01:04:48.2682929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2683038Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2683041Z 2025-10-10T01:04:48.2683144Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2683343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2683418Z res = mod(**inputs) 2025-10-10T01:04:48.2683673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2683767Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2684023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2684093Z outputs = layer_module( 2025-10-10T01:04:48.2684354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2684423Z outputs = self.rel_attn( 2025-10-10T01:04:48.2684683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2684757Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2685027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2685158Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2685163Z 2025-10-10T01:04:48.2685269Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2685475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2685540Z res = mod(**inputs) 2025-10-10T01:04:48.2685822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2685905Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2686162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2686239Z outputs = layer_module( 2025-10-10T01:04:48.2686495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2686570Z outputs = self.rel_attn( 2025-10-10T01:04:48.2686842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2686936Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2687217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2687335Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2687339Z 2025-10-10T01:04:48.2687449Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2687650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2687724Z res = mod(**inputs) 2025-10-10T01:04:48.2687980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2688064Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2688346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2688435Z outputs = layer_module( 2025-10-10T01:04:48.2688700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2688771Z outputs = self.rel_attn( 2025-10-10T01:04:48.2689028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2689126Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2689403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2689523Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2689527Z 2025-10-10T01:04:48.2689628Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2689836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2689905Z res = mod(**inputs) 2025-10-10T01:04:48.2690161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2690252Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2690507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2690583Z outputs = layer_module( 2025-10-10T01:04:48.2690839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2691045Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2691310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2691394Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2691659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2691733Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2691992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2692092Z output = self.layer_1(output) 2025-10-10T01:04:48.2692095Z 2025-10-10T01:04:48.2692200Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2692405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2692472Z res = mod(**inputs) 2025-10-10T01:04:48.2692739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2692822Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2693095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2693173Z outputs = layer_module( 2025-10-10T01:04:48.2693428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2693651Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2693933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2694024Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2694298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2694376Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2694671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2694811Z output = self.activation_function(output) 2025-10-10T01:04:48.2695049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2695122Z return self.act(input) 2025-10-10T01:04:48.2695125Z 2025-10-10T01:04:48.2695231Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2695440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2695508Z res = mod(**inputs) 2025-10-10T01:04:48.2695778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2695862Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2696125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2696208Z outputs = layer_module( 2025-10-10T01:04:48.2696470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2696684Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2696955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2697042Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2697304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2697379Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2697645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2697720Z output = self.layer_2(output) 2025-10-10T01:04:48.2697727Z 2025-10-10T01:04:48.2697842Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2698046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2698115Z res = mod(**inputs) 2025-10-10T01:04:48.2698398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2698484Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2698764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2698838Z outputs = layer_module( 2025-10-10T01:04:48.2699125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2699196Z outputs = self.rel_attn( 2025-10-10T01:04:48.2699467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2699579Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2699583Z 2025-10-10T01:04:48.2699685Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2699891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2699957Z res = mod(**inputs) 2025-10-10T01:04:48.2700212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2700302Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2700557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2700633Z outputs = layer_module( 2025-10-10T01:04:48.2700908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2701003Z outputs = self.rel_attn( 2025-10-10T01:04:48.2701258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2701363Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2701368Z 2025-10-10T01:04:48.2701480Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2701680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2701754Z res = mod(**inputs) 2025-10-10T01:04:48.2702010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2702094Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2702356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2702431Z outputs = layer_module( 2025-10-10T01:04:48.2702700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2702773Z outputs = self.rel_attn( 2025-10-10T01:04:48.2703045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2703134Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2703422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2703570Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2703574Z 2025-10-10T01:04:48.2703684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2703902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2703974Z res = mod(**inputs) 2025-10-10T01:04:48.2704252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2704347Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2704619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2704718Z outputs = layer_module( 2025-10-10T01:04:48.2704990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2705063Z outputs = self.rel_attn( 2025-10-10T01:04:48.2705338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2705480Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2705484Z 2025-10-10T01:04:48.2705617Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2705832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2705908Z res = mod(**inputs) 2025-10-10T01:04:48.2706269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2706377Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2706665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2706743Z outputs = layer_module( 2025-10-10T01:04:48.2707029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2707105Z outputs = self.rel_attn( 2025-10-10T01:04:48.2707385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2707526Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2707813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2707958Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2707963Z 2025-10-10T01:04:48.2708073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2708295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2708366Z res = mod(**inputs) 2025-10-10T01:04:48.2708643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2708738Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2709016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2709099Z outputs = layer_module( 2025-10-10T01:04:48.2709374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2709448Z outputs = self.rel_attn( 2025-10-10T01:04:48.2709729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2709834Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2709838Z 2025-10-10T01:04:48.2709953Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2710165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2710235Z res = mod(**inputs) 2025-10-10T01:04:48.2710522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2710610Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2710901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2710976Z outputs = layer_module( 2025-10-10T01:04:48.2711260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2711355Z outputs = self.rel_attn( 2025-10-10T01:04:48.2711622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2711710Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2712004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2712143Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2712147Z 2025-10-10T01:04:48.2712258Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2712495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2712577Z res = mod(**inputs) 2025-10-10T01:04:48.2712849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2712950Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2713225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2713309Z outputs = layer_module( 2025-10-10T01:04:48.2713585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2713664Z outputs = self.rel_attn( 2025-10-10T01:04:48.2713945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2714086Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2714389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2714509Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2714515Z 2025-10-10T01:04:48.2714625Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2714844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2714914Z res = mod(**inputs) 2025-10-10T01:04:48.2715193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2715288Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2715568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2715647Z outputs = layer_module( 2025-10-10T01:04:48.2715921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2716004Z outputs = self.rel_attn( 2025-10-10T01:04:48.2716274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2716379Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2716672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2716792Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2716796Z 2025-10-10T01:04:48.2716913Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2717126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2717203Z res = mod(**inputs) 2025-10-10T01:04:48.2717481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2717570Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2717851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2717943Z outputs = layer_module( 2025-10-10T01:04:48.2718219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2718437Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2718724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2718808Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2719098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2719189Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2719463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2719552Z output = self.layer_1(output) 2025-10-10T01:04:48.2719556Z 2025-10-10T01:04:48.2719667Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2719887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2719958Z res = mod(**inputs) 2025-10-10T01:04:48.2720232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2720327Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2720622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2720719Z outputs = layer_module( 2025-10-10T01:04:48.2720990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2721198Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2721467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2721547Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2721811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2721885Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2722137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2722240Z output = self.activation_function(output) 2025-10-10T01:04:48.2722456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2722535Z return self.act(input) 2025-10-10T01:04:48.2722538Z 2025-10-10T01:04:48.2722643Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2722848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2722914Z res = mod(**inputs) 2025-10-10T01:04:48.2723170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2723259Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2723515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2723589Z outputs = layer_module( 2025-10-10T01:04:48.2723849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2724056Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2724341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2724419Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2724687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2724765Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2725039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2725118Z output = self.layer_2(output) 2025-10-10T01:04:48.2725121Z 2025-10-10T01:04:48.2725255Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2725482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2725554Z res = mod(**inputs) 2025-10-10T01:04:48.2725834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2725926Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2726199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2726278Z outputs = layer_module( 2025-10-10T01:04:48.2726549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2726633Z outputs = self.rel_attn( 2025-10-10T01:04:48.2726902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2727051Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2727055Z 2025-10-10T01:04:48.2727174Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2727374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2727451Z res = mod(**inputs) 2025-10-10T01:04:48.2727708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2727801Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2728058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2728127Z outputs = layer_module( 2025-10-10T01:04:48.2728391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2728461Z outputs = self.rel_attn( 2025-10-10T01:04:48.2728726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2728829Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2728833Z 2025-10-10T01:04:48.2728945Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2729147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2729213Z res = mod(**inputs) 2025-10-10T01:04:48.2729476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2729560Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2729820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2729889Z outputs = layer_module( 2025-10-10T01:04:48.2730147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2730225Z outputs = self.rel_attn( 2025-10-10T01:04:48.2730480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2730582Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2730860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2730992Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2731003Z 2025-10-10T01:04:48.2731109Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2731312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2731385Z res = mod(**inputs) 2025-10-10T01:04:48.2731847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2731950Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2732209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2732283Z outputs = layer_module( 2025-10-10T01:04:48.2732549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2732620Z outputs = self.rel_attn( 2025-10-10T01:04:48.2732885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2733022Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2733026Z 2025-10-10T01:04:48.2733129Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2733364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2733455Z res = mod(**inputs) 2025-10-10T01:04:48.2733722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2733809Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2734072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2734142Z outputs = layer_module( 2025-10-10T01:04:48.2734397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2734476Z outputs = self.rel_attn( 2025-10-10T01:04:48.2734730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2734814Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2735090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2735222Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2735225Z 2025-10-10T01:04:48.2735340Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2735540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2735615Z res = mod(**inputs) 2025-10-10T01:04:48.2735874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2735958Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2736223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2736290Z outputs = layer_module( 2025-10-10T01:04:48.2736543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2736609Z outputs = self.rel_attn( 2025-10-10T01:04:48.2736856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2736973Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2736976Z 2025-10-10T01:04:48.2737073Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2737269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2737333Z res = mod(**inputs) 2025-10-10T01:04:48.2737581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2737660Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2737917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2737994Z outputs = layer_module( 2025-10-10T01:04:48.2738232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2738306Z outputs = self.rel_attn( 2025-10-10T01:04:48.2738545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2738622Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2738879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2738995Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2738999Z 2025-10-10T01:04:48.2739104Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2739315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2739401Z res = mod(**inputs) 2025-10-10T01:04:48.2739645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2739725Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2739977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2740043Z outputs = layer_module( 2025-10-10T01:04:48.2740293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2740358Z outputs = self.rel_attn( 2025-10-10T01:04:48.2740601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2740694Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2740962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2741077Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2741080Z 2025-10-10T01:04:48.2741181Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2741378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2741440Z res = mod(**inputs) 2025-10-10T01:04:48.2741685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2741771Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2742014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2742087Z outputs = layer_module( 2025-10-10T01:04:48.2742331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2742399Z outputs = self.rel_attn( 2025-10-10T01:04:48.2742648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2742753Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2743019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2743126Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2743129Z 2025-10-10T01:04:48.2743236Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2743426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2743491Z res = mod(**inputs) 2025-10-10T01:04:48.2743759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2743842Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2744096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2744162Z outputs = layer_module( 2025-10-10T01:04:48.2744403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2744607Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2744856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2744941Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2745210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2745310Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2745559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2745632Z output = self.layer_1(output) 2025-10-10T01:04:48.2745638Z 2025-10-10T01:04:48.2745751Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2745946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2746018Z res = mod(**inputs) 2025-10-10T01:04:48.2746321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2746408Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2746666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2746739Z outputs = layer_module( 2025-10-10T01:04:48.2746993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2747195Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2747458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2747536Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2747786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2747866Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2748122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2748216Z output = self.activation_function(output) 2025-10-10T01:04:48.2748428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2748497Z return self.act(input) 2025-10-10T01:04:48.2748500Z 2025-10-10T01:04:48.2748607Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2748820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2748891Z res = mod(**inputs) 2025-10-10T01:04:48.2749140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2749229Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2749479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2749546Z outputs = layer_module( 2025-10-10T01:04:48.2749825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2750023Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2750279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2750356Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2750602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2750679Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2750921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2751000Z output = self.layer_2(output) 2025-10-10T01:04:48.2751003Z 2025-10-10T01:04:48.2751101Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2751315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2751394Z res = mod(**inputs) 2025-10-10T01:04:48.2751640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2751730Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2751974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2752044Z outputs = layer_module( 2025-10-10T01:04:48.2752288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2752354Z outputs = self.rel_attn( 2025-10-10T01:04:48.2752601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2752694Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2752701Z 2025-10-10T01:04:48.2752805Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2752996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2753060Z res = mod(**inputs) 2025-10-10T01:04:48.2753314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2753393Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2753644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2753709Z outputs = layer_module( 2025-10-10T01:04:48.2753956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2754022Z outputs = self.rel_attn( 2025-10-10T01:04:48.2754268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2754372Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2754376Z 2025-10-10T01:04:48.2754473Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2754685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2754749Z res = mod(**inputs) 2025-10-10T01:04:48.2754991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2755078Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2755321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2755394Z outputs = layer_module( 2025-10-10T01:04:48.2755649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2755728Z outputs = self.rel_attn( 2025-10-10T01:04:48.2755969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2756041Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2756305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2756432Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2756436Z 2025-10-10T01:04:48.2756544Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2756734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2756800Z res = mod(**inputs) 2025-10-10T01:04:48.2757067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2757178Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2757432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2757500Z outputs = layer_module( 2025-10-10T01:04:48.2757749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2757825Z outputs = self.rel_attn( 2025-10-10T01:04:48.2758076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2758222Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2758225Z 2025-10-10T01:04:48.2758323Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2758520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2758588Z res = mod(**inputs) 2025-10-10T01:04:48.2758835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2758921Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2759165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2759237Z outputs = layer_module( 2025-10-10T01:04:48.2759489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2759553Z outputs = self.rel_attn( 2025-10-10T01:04:48.2759795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2759865Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2760124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2760247Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2760249Z 2025-10-10T01:04:48.2760351Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2760549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2760610Z res = mod(**inputs) 2025-10-10T01:04:48.2760852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2760929Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2761178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2761244Z outputs = layer_module( 2025-10-10T01:04:48.2761512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2761589Z outputs = self.rel_attn( 2025-10-10T01:04:48.2761834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2761935Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2761939Z 2025-10-10T01:04:48.2762034Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2762225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2762288Z res = mod(**inputs) 2025-10-10T01:04:48.2762534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2762620Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2762905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2763010Z outputs = layer_module( 2025-10-10T01:04:48.2763246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2763313Z outputs = self.rel_attn( 2025-10-10T01:04:48.2763556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2763625Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2763883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2763998Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2764002Z 2025-10-10T01:04:48.2764106Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2764290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2764356Z res = mod(**inputs) 2025-10-10T01:04:48.2764600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2764677Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2764919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2764982Z outputs = layer_module( 2025-10-10T01:04:48.2765216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2765290Z outputs = self.rel_attn( 2025-10-10T01:04:48.2765523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2765615Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2765875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2765984Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2765995Z 2025-10-10T01:04:48.2766094Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2766304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2766374Z res = mod(**inputs) 2025-10-10T01:04:48.2766622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2766707Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2766954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2767018Z outputs = layer_module( 2025-10-10T01:04:48.2767291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2767359Z outputs = self.rel_attn( 2025-10-10T01:04:48.2767610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2767696Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2767958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2768074Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2768077Z 2025-10-10T01:04:48.2768175Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2768375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2768438Z res = mod(**inputs) 2025-10-10T01:04:48.2768680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2768792Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2769029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2769103Z outputs = layer_module( 2025-10-10T01:04:48.2769339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2769537Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2769781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2769856Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2770098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2770167Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2770415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2770485Z output = self.layer_1(output) 2025-10-10T01:04:48.2770489Z 2025-10-10T01:04:48.2770593Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2770778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2770841Z res = mod(**inputs) 2025-10-10T01:04:48.2771085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2771163Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2771408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2771472Z outputs = layer_module( 2025-10-10T01:04:48.2771710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2771909Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2772152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2772251Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2772490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2772566Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2772803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2772889Z output = self.activation_function(output) 2025-10-10T01:04:48.2773115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2773185Z return self.act(input) 2025-10-10T01:04:48.2773188Z 2025-10-10T01:04:48.2773292Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2773477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2773540Z res = mod(**inputs) 2025-10-10T01:04:48.2773784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2773861Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2774104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2774169Z outputs = layer_module( 2025-10-10T01:04:48.2774411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2775006Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2775252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2775336Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2775571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2775647Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2775883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2775953Z output = self.layer_2(output) 2025-10-10T01:04:48.2775956Z 2025-10-10T01:04:48.2776063Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2776249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2776323Z res = mod(**inputs) 2025-10-10T01:04:48.2776562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2776640Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2776887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2776953Z outputs = layer_module( 2025-10-10T01:04:48.2777196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2777262Z outputs = self.rel_attn( 2025-10-10T01:04:48.2777505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2777598Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2777601Z 2025-10-10T01:04:48.2777702Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2777896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2777957Z res = mod(**inputs) 2025-10-10T01:04:48.2778202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2778301Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2778542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2778616Z outputs = layer_module( 2025-10-10T01:04:48.2778855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2778928Z outputs = self.rel_attn( 2025-10-10T01:04:48.2779167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2779287Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2779291Z 2025-10-10T01:04:48.2779387Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2779569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2779642Z res = mod(**inputs) 2025-10-10T01:04:48.2779879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2779964Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2780201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2780265Z outputs = layer_module( 2025-10-10T01:04:48.2780510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2780607Z outputs = self.rel_attn( 2025-10-10T01:04:48.2780850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2780920Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2781175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2781306Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2781309Z 2025-10-10T01:04:48.2781405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2781596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2781657Z res = mod(**inputs) 2025-10-10T01:04:48.2781904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2781981Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2782226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2782300Z outputs = layer_module( 2025-10-10T01:04:48.2782540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2782615Z outputs = self.rel_attn( 2025-10-10T01:04:48.2782854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2782982Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2782991Z 2025-10-10T01:04:48.2783089Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2783276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2783346Z res = mod(**inputs) 2025-10-10T01:04:48.2783594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2783681Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2783923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2784014Z outputs = layer_module( 2025-10-10T01:04:48.2784267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2784335Z outputs = self.rel_attn( 2025-10-10T01:04:48.2784601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2784672Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2784937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2785085Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2785089Z 2025-10-10T01:04:48.2785192Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2785392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2785459Z res = mod(**inputs) 2025-10-10T01:04:48.2785723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2785805Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2786058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2786134Z outputs = layer_module( 2025-10-10T01:04:48.2786457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2786578Z outputs = self.rel_attn( 2025-10-10T01:04:48.2786828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2786926Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2786932Z 2025-10-10T01:04:48.2787052Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2787269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2787352Z res = mod(**inputs) 2025-10-10T01:04:48.2787631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2787723Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2787992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2788062Z outputs = layer_module( 2025-10-10T01:04:48.2788326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2788407Z outputs = self.rel_attn( 2025-10-10T01:04:48.2788664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2788738Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2788998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2789125Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2789128Z 2025-10-10T01:04:48.2789226Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2789420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2789483Z res = mod(**inputs) 2025-10-10T01:04:48.2789728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2789816Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2790058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2790148Z outputs = layer_module( 2025-10-10T01:04:48.2790399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2790473Z outputs = self.rel_attn( 2025-10-10T01:04:48.2790721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2790808Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2791081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2791206Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2791209Z 2025-10-10T01:04:48.2791317Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2791506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2791572Z res = mod(**inputs) 2025-10-10T01:04:48.2791824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2791905Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2792161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2792227Z outputs = layer_module( 2025-10-10T01:04:48.2792474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2792541Z outputs = self.rel_attn( 2025-10-10T01:04:48.2792842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2792940Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2793199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2793315Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2793319Z 2025-10-10T01:04:48.2793418Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2793605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2793676Z res = mod(**inputs) 2025-10-10T01:04:48.2793920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2794005Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2794250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2794317Z outputs = layer_module( 2025-10-10T01:04:48.2794565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2794764Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2795022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2795098Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2795349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2795422Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2795666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2795748Z output = self.layer_1(output) 2025-10-10T01:04:48.2795751Z 2025-10-10T01:04:48.2795849Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2796062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2796125Z res = mod(**inputs) 2025-10-10T01:04:48.2796371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2796456Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2796700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2796772Z outputs = layer_module( 2025-10-10T01:04:48.2797030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2797236Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2797484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2797560Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2797808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2797879Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2798127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2798212Z output = self.activation_function(output) 2025-10-10T01:04:48.2798417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2798525Z return self.act(input) 2025-10-10T01:04:48.2798529Z 2025-10-10T01:04:48.2798629Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2798826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2798891Z res = mod(**inputs) 2025-10-10T01:04:48.2799143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2799223Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2799466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2799540Z outputs = layer_module( 2025-10-10T01:04:48.2799780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2799988Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2800238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2800314Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2800570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2800640Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2800888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2800960Z output = self.layer_2(output) 2025-10-10T01:04:48.2800963Z 2025-10-10T01:04:48.2801069Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2801258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2801323Z res = mod(**inputs) 2025-10-10T01:04:48.2801579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2801659Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2801909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2801994Z outputs = layer_module( 2025-10-10T01:04:48.2802236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2802311Z outputs = self.rel_attn( 2025-10-10T01:04:48.2802554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-10-10T01:04:48.2802656Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-10-10T01:04:48.2802660Z 2025-10-10T01:04:48.2802759Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2802974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2803040Z res = mod(**inputs) 2025-10-10T01:04:48.2803284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2803371Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2803616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2803688Z outputs = layer_module( 2025-10-10T01:04:48.2803932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2804000Z outputs = self.rel_attn( 2025-10-10T01:04:48.2804251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-10-10T01:04:48.2804364Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-10-10T01:04:48.2804387Z 2025-10-10T01:04:48.2804494Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2804684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2804749Z res = mod(**inputs) 2025-10-10T01:04:48.2805002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2805081Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2805333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2805398Z outputs = layer_module( 2025-10-10T01:04:48.2805646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2805713Z outputs = self.rel_attn( 2025-10-10T01:04:48.2805959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2806039Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2806299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-10-10T01:04:48.2806433Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-10-10T01:04:48.2806437Z 2025-10-10T01:04:48.2806537Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2806727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2806797Z res = mod(**inputs) 2025-10-10T01:04:48.2807043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2807128Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2807373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2807448Z outputs = layer_module( 2025-10-10T01:04:48.2807689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2807776Z outputs = self.rel_attn( 2025-10-10T01:04:48.2808031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-10-10T01:04:48.2808165Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-10-10T01:04:48.2808168Z 2025-10-10T01:04:48.2808278Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2808472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2808536Z res = mod(**inputs) 2025-10-10T01:04:48.2808819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2808902Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2809161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2809233Z outputs = layer_module( 2025-10-10T01:04:48.2809494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2809563Z outputs = self.rel_attn( 2025-10-10T01:04:48.2809823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2809902Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2810168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-10-10T01:04:48.2810318Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-10-10T01:04:48.2810334Z 2025-10-10T01:04:48.2810436Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2810629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2810703Z res = mod(**inputs) 2025-10-10T01:04:48.2810951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2811039Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2811286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2811353Z outputs = layer_module( 2025-10-10T01:04:48.2811607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2811676Z outputs = self.rel_attn( 2025-10-10T01:04:48.2811932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-10-10T01:04:48.2812032Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-10-10T01:04:48.2812035Z 2025-10-10T01:04:48.2812140Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2812338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2812402Z res = mod(**inputs) 2025-10-10T01:04:48.2812658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2812739Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2813001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2813069Z outputs = layer_module( 2025-10-10T01:04:48.2813326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2813404Z outputs = self.rel_attn( 2025-10-10T01:04:48.2813657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-10-10T01:04:48.2813753Z attn_vec = self.rel_attn_core( 2025-10-10T01:04:48.2814022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-10-10T01:04:48.2814151Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-10-10T01:04:48.2814155Z 2025-10-10T01:04:48.2814257Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2814454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2814528Z res = mod(**inputs) 2025-10-10T01:04:48.2814813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2814913Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2815187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2815273Z outputs = layer_module( 2025-10-10T01:04:48.2815536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2815606Z outputs = self.rel_attn( 2025-10-10T01:04:48.2815868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2815961Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2816236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2816357Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2816393Z 2025-10-10T01:04:48.2816499Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2816705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2816773Z res = mod(**inputs) 2025-10-10T01:04:48.2817039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2817122Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2817378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2817455Z outputs = layer_module( 2025-10-10T01:04:48.2817712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-10-10T01:04:48.2817793Z outputs = self.rel_attn( 2025-10-10T01:04:48.2818062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-10-10T01:04:48.2818159Z output_h = self.post_attention(h, attn_vec) 2025-10-10T01:04:48.2818459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-10-10T01:04:48.2818580Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-10-10T01:04:48.2818584Z 2025-10-10T01:04:48.2818700Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2818911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2818988Z res = mod(**inputs) 2025-10-10T01:04:48.2819259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2819347Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2819628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2819703Z outputs = layer_module( 2025-10-10T01:04:48.2819979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2820214Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2820494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2820587Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2820859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2820946Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2821214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-10-10T01:04:48.2821319Z output = self.layer_1(output) 2025-10-10T01:04:48.2821323Z 2025-10-10T01:04:48.2821432Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2821640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2821719Z res = mod(**inputs) 2025-10-10T01:04:48.2821992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2822089Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2822359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2822432Z outputs = layer_module( 2025-10-10T01:04:48.2822707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2822945Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2823250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2823336Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2823615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2823695Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2823967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-10-10T01:04:48.2824071Z output = self.activation_function(output) 2025-10-10T01:04:48.2824301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:04:48.2824384Z return self.act(input) 2025-10-10T01:04:48.2824387Z 2025-10-10T01:04:48.2824503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2824716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2824794Z res = mod(**inputs) 2025-10-10T01:04:48.2825068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-10-10T01:04:48.2825165Z transformer_outputs = self.transformer( 2025-10-10T01:04:48.2825436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-10-10T01:04:48.2825515Z outputs = layer_module( 2025-10-10T01:04:48.2825787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-10-10T01:04:48.2826008Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-10-10T01:04:48.2826390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:04:48.2826482Z return forward_fn(*input_tensors) 2025-10-10T01:04:48.2826771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-10-10T01:04:48.2826873Z output_x = self.ff(output_x) 2025-10-10T01:04:48.2827154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-10-10T01:04:48.2827245Z output = self.layer_2(output) 2025-10-10T01:04:48.2827249Z 2025-10-10T01:04:48.2827363Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2827599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2827668Z res = mod(**inputs) 2025-10-10T01:04:48.2827969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-10-10T01:04:48.2828080Z logits = self.lm_loss(transformer_outputs[0]) 2025-10-10T01:04:48.2828085Z 2025-10-10T01:04:48.2828197Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:04:48.2828419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:04:48.2828491Z res = mod(**inputs) 2025-10-10T01:04:48.2828770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-10-10T01:04:48.2828915Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-10-10T01:04:48.2828918Z 2025-10-10T01:05:02.2772962Z Compilation time (from dynamo_timed): 34.072005965 2025-10-10T01:05:02.2813475Z pass 2025-10-10T01:05:02.2814380Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:05:02.2815520Z TIMING: _recursive_pre_grad_passes:0.01236 _recursive_joint_graph_passes:1.48515 _recursive_post_grad_passes:0.22892 async_compile.wait:1.0208 code_gen:12.7405 inductor_compile:18.03152 backend_compile:27.803 gc:0.00309 entire_frame_compile:34.07201 total_wall_time:34.07201 2025-10-10T01:05:02.2816704Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:33127 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:6846 2025-10-10T01:05:02.2817288Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-10-10T01:05:05.8697174Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-10-10T01:05:05.8698047Z import pynvml # type: ignore[import] 2025-10-10T01:05:09.7751385Z 2025-10-10T01:05:11.0292838Z loading model: 0it [00:00, ?it/s] 2025-10-10T01:05:11.0293203Z loading model: 0it [00:01, ?it/s] 2025-10-10T01:05:11.0311567Z cpu eval YituTechConvBert 2025-10-10T01:05:12.0337354Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:05:12.3171159Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:05:12.5905776Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:05:25.5902967Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5903434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5903786Z res = mod(**inputs) 2025-10-10T01:05:25.5904246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5904711Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5905217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5905658Z hidden_states = self.encoder( 2025-10-10T01:05:25.5906098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5907003Z layer_outputs = layer_module( 2025-10-10T01:05:25.5907424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5907853Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5908299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5908752Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5909273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5909728Z self_outputs = self.self( 2025-10-10T01:05:25.5910152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.5910606Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.5910786Z 2025-10-10T01:05:25.5910908Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5911320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5911684Z res = mod(**inputs) 2025-10-10T01:05:25.5912099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5912554Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5913005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5913568Z hidden_states = self.encoder( 2025-10-10T01:05:25.5914007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5914444Z layer_outputs = layer_module( 2025-10-10T01:05:25.5914841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5915344Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5915778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5916228Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5916693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5917138Z self_outputs = self.self( 2025-10-10T01:05:25.5917579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.5918026Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.5918178Z 2025-10-10T01:05:25.5918302Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5918688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5919043Z res = mod(**inputs) 2025-10-10T01:05:25.5919454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5919898Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5920336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5920755Z hidden_states = self.encoder( 2025-10-10T01:05:25.5921185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5921621Z layer_outputs = layer_module( 2025-10-10T01:05:25.5921994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5922408Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5922842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5923301Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5923742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5924172Z self_outputs = self.self( 2025-10-10T01:05:25.5924589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.5925075Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.5925245Z 2025-10-10T01:05:25.5925339Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.5925577Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.5925838Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5926237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5926597Z res = mod(**inputs) 2025-10-10T01:05:25.5927004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5927501Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5927954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5928398Z hidden_states = self.encoder( 2025-10-10T01:05:25.5928867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5929316Z layer_outputs = layer_module( 2025-10-10T01:05:25.5929696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5930083Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5930517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5930953Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5931391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5932013Z self_outputs = self.self( 2025-10-10T01:05:25.5932428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.5932896Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.5933076Z 2025-10-10T01:05:25.5933166Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.5933429Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5933814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5934167Z res = mod(**inputs) 2025-10-10T01:05:25.5934577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5935026Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5935470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5935898Z hidden_states = self.encoder( 2025-10-10T01:05:25.5936327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5936764Z layer_outputs = layer_module( 2025-10-10T01:05:25.5937139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5937522Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5937994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5938426Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5938841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5939247Z self_outputs = self.self( 2025-10-10T01:05:25.5939632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.5940154Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.5940749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.5941194Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.5941339Z 2025-10-10T01:05:25.5941467Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5941861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5942223Z res = mod(**inputs) 2025-10-10T01:05:25.5942654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5943113Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5943560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5944000Z hidden_states = self.encoder( 2025-10-10T01:05:25.5944472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5944928Z layer_outputs = layer_module( 2025-10-10T01:05:25.5945302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5945686Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5946133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5946738Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5947198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5947627Z self_outputs = self.self( 2025-10-10T01:05:25.5948027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.5948535Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.5949035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.5949451Z x = self.pointwise(x) 2025-10-10T01:05:25.5949567Z 2025-10-10T01:05:25.5949684Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5950049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5950384Z res = mod(**inputs) 2025-10-10T01:05:25.5950767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5951194Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5951615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5952051Z hidden_states = self.encoder( 2025-10-10T01:05:25.5952450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5952854Z layer_outputs = layer_module( 2025-10-10T01:05:25.5953272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5953645Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5954067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5954498Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5954944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5955377Z self_outputs = self.self( 2025-10-10T01:05:25.5955814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.5956347Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.5956583Z 2025-10-10T01:05:25.5956693Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5957068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5957401Z res = mod(**inputs) 2025-10-10T01:05:25.5957786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5958209Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5958626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5959049Z hidden_states = self.encoder( 2025-10-10T01:05:25.5959487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5959932Z layer_outputs = layer_module( 2025-10-10T01:05:25.5960303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5960694Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5961125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5961553Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5961988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5962417Z self_outputs = self.self( 2025-10-10T01:05:25.5962836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.5963325Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.5963514Z 2025-10-10T01:05:25.5963629Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5964016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5964367Z res = mod(**inputs) 2025-10-10T01:05:25.5964773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5965204Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5965643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5966075Z hidden_states = self.encoder( 2025-10-10T01:05:25.5966498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5966934Z layer_outputs = layer_module( 2025-10-10T01:05:25.5967298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5967688Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5968119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5968581Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5969030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5969453Z self_outputs = self.self( 2025-10-10T01:05:25.5969864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.5970336Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.5970532Z 2025-10-10T01:05:25.5970652Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.5970888Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.5971141Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5971532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5971884Z res = mod(**inputs) 2025-10-10T01:05:25.5972269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5972683Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5973098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5973501Z hidden_states = self.encoder( 2025-10-10T01:05:25.5973927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5974396Z layer_outputs = layer_module( 2025-10-10T01:05:25.5974762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5975151Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5975585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5975999Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5976406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.5976817Z self_outputs = self.self( 2025-10-10T01:05:25.5977210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.5977682Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.5977860Z 2025-10-10T01:05:25.5977988Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5978368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5978716Z res = mod(**inputs) 2025-10-10T01:05:25.5979100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5979520Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5979934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5980336Z hidden_states = self.encoder( 2025-10-10T01:05:25.5980761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5981189Z layer_outputs = layer_module( 2025-10-10T01:05:25.5981567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5981964Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5982405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.5982875Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.5983361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.5983883Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.5984393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.5984861Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.5985023Z 2025-10-10T01:05:25.5985139Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5985557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5985920Z res = mod(**inputs) 2025-10-10T01:05:25.5986439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5986917Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5987385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5987843Z hidden_states = self.encoder( 2025-10-10T01:05:25.5988293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5988738Z layer_outputs = layer_module( 2025-10-10T01:05:25.5989131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5989536Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5990042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.5990496Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.5990949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.5991394Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.5991875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.5992409Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.5992898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.5993353Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.5993517Z 2025-10-10T01:05:25.5993637Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.5994033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.5994391Z res = mod(**inputs) 2025-10-10T01:05:25.5994802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.5995258Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.5995711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.5996154Z hidden_states = self.encoder( 2025-10-10T01:05:25.5996588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.5997019Z layer_outputs = layer_module( 2025-10-10T01:05:25.5997406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.5997810Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.5998258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.5998727Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.5999178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.5999619Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6000105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6000624Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6001101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6001595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6002015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6002392Z return self.act(input) 2025-10-10T01:05:25.6002520Z 2025-10-10T01:05:25.6002648Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6003043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6003403Z res = mod(**inputs) 2025-10-10T01:05:25.6003837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6004289Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6004746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6005183Z hidden_states = self.encoder( 2025-10-10T01:05:25.6005654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6006096Z layer_outputs = layer_module( 2025-10-10T01:05:25.6006472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6006870Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6007333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6007795Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6008252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6008680Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6009145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6009684Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6010194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6010639Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6010790Z 2025-10-10T01:05:25.6010911Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6011294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6011646Z res = mod(**inputs) 2025-10-10T01:05:25.6012051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6012502Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6012941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6013380Z hidden_states = self.encoder( 2025-10-10T01:05:25.6013813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6014272Z layer_outputs = layer_module( 2025-10-10T01:05:25.6014655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6015058Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6015491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6015932Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6016372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6016819Z self_outputs = self.self( 2025-10-10T01:05:25.6017237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6017702Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6017876Z 2025-10-10T01:05:25.6017995Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6018395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6018759Z res = mod(**inputs) 2025-10-10T01:05:25.6019172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6019631Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6020087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6020535Z hidden_states = self.encoder( 2025-10-10T01:05:25.6021003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6021447Z layer_outputs = layer_module( 2025-10-10T01:05:25.6021834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6022240Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6022688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6023138Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6023610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6024051Z self_outputs = self.self( 2025-10-10T01:05:25.6024486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6024946Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6025101Z 2025-10-10T01:05:25.6025218Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6025616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6025977Z res = mod(**inputs) 2025-10-10T01:05:25.6026489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6026948Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6027403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6027851Z hidden_states = self.encoder( 2025-10-10T01:05:25.6028285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6028735Z layer_outputs = layer_module( 2025-10-10T01:05:25.6029116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6029522Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6029998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6030450Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6030904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6031340Z self_outputs = self.self( 2025-10-10T01:05:25.6032031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6032507Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6032679Z 2025-10-10T01:05:25.6032843Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6033065Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6033317Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6033688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6034030Z res = mod(**inputs) 2025-10-10T01:05:25.6034426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6034850Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6035273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6035688Z hidden_states = self.encoder( 2025-10-10T01:05:25.6036099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6036557Z layer_outputs = layer_module( 2025-10-10T01:05:25.6036913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6037287Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6037705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6038122Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6038539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6038955Z self_outputs = self.self( 2025-10-10T01:05:25.6039357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6039808Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6039977Z 2025-10-10T01:05:25.6040066Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6040297Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6040655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6040974Z res = mod(**inputs) 2025-10-10T01:05:25.6041343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6041742Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6042147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6042549Z hidden_states = self.encoder( 2025-10-10T01:05:25.6042946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6043347Z layer_outputs = layer_module( 2025-10-10T01:05:25.6043704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6044086Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6044511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6044960Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6045378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6045770Z self_outputs = self.self( 2025-10-10T01:05:25.6046146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6046678Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6047189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6047598Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6047730Z 2025-10-10T01:05:25.6047838Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6048197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6048527Z res = mod(**inputs) 2025-10-10T01:05:25.6048889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6049281Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6049682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6050078Z hidden_states = self.encoder( 2025-10-10T01:05:25.6050489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6050908Z layer_outputs = layer_module( 2025-10-10T01:05:25.6051236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6051586Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6051991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6052403Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6052818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6053216Z self_outputs = self.self( 2025-10-10T01:05:25.6053612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6054110Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6054610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6055017Z x = self.pointwise(x) 2025-10-10T01:05:25.6055133Z 2025-10-10T01:05:25.6055244Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6055626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6055973Z res = mod(**inputs) 2025-10-10T01:05:25.6056367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6056782Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6057226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6057653Z hidden_states = self.encoder( 2025-10-10T01:05:25.6058060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6058468Z layer_outputs = layer_module( 2025-10-10T01:05:25.6058814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6059207Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6059618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6060038Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6060455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6060875Z self_outputs = self.self( 2025-10-10T01:05:25.6061313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6061833Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6062057Z 2025-10-10T01:05:25.6062178Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6062564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6062913Z res = mod(**inputs) 2025-10-10T01:05:25.6063332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6063788Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6064237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6064669Z hidden_states = self.encoder( 2025-10-10T01:05:25.6065139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6065590Z layer_outputs = layer_module( 2025-10-10T01:05:25.6065964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6066435Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6066874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6067317Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6067773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6068214Z self_outputs = self.self( 2025-10-10T01:05:25.6068637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6069099Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6069291Z 2025-10-10T01:05:25.6069402Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6069774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6070107Z res = mod(**inputs) 2025-10-10T01:05:25.6070489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6070913Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6071334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6071743Z hidden_states = self.encoder( 2025-10-10T01:05:25.6072145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6072546Z layer_outputs = layer_module( 2025-10-10T01:05:25.6072911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6073286Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6073703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6074142Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6074548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6074957Z self_outputs = self.self( 2025-10-10T01:05:25.6075357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6075822Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6076008Z 2025-10-10T01:05:25.6076092Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6076335Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6076582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6076950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6077299Z res = mod(**inputs) 2025-10-10T01:05:25.6077700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6078142Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6078580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6078988Z hidden_states = self.encoder( 2025-10-10T01:05:25.6079382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6079789Z layer_outputs = layer_module( 2025-10-10T01:05:25.6080187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6080580Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6081010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6081423Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6081837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6082243Z self_outputs = self.self( 2025-10-10T01:05:25.6082647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6083120Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6083297Z 2025-10-10T01:05:25.6083411Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6083803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6084151Z res = mod(**inputs) 2025-10-10T01:05:25.6084554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6084993Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6085434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6085869Z hidden_states = self.encoder( 2025-10-10T01:05:25.6086293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6086734Z layer_outputs = layer_module( 2025-10-10T01:05:25.6087100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6087499Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6087936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6088390Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6088848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6089347Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6089849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6090286Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6090428Z 2025-10-10T01:05:25.6090542Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6090908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6091258Z res = mod(**inputs) 2025-10-10T01:05:25.6091643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6092063Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6092484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6092917Z hidden_states = self.encoder( 2025-10-10T01:05:25.6093343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6093751Z layer_outputs = layer_module( 2025-10-10T01:05:25.6094111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6094483Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6094908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6095357Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6095797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6096239Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6096720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6097247Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6097744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6098178Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6098324Z 2025-10-10T01:05:25.6098440Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6098818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6099149Z res = mod(**inputs) 2025-10-10T01:05:25.6099540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6099971Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6100398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6100812Z hidden_states = self.encoder( 2025-10-10T01:05:25.6101242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6101688Z layer_outputs = layer_module( 2025-10-10T01:05:25.6102075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6102484Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6102922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6103381Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6103837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6104261Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6104726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6105237Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6105726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6106297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6106727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6107118Z return self.act(input) 2025-10-10T01:05:25.6107243Z 2025-10-10T01:05:25.6107360Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6107752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6108103Z res = mod(**inputs) 2025-10-10T01:05:25.6108511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6108947Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6109404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6109844Z hidden_states = self.encoder( 2025-10-10T01:05:25.6110308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6110751Z layer_outputs = layer_module( 2025-10-10T01:05:25.6111119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6111514Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6111951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6112405Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6112836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6113264Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6113729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6114262Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6114758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6115198Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6115355Z 2025-10-10T01:05:25.6115470Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6115857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6116203Z res = mod(**inputs) 2025-10-10T01:05:25.6116605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6117039Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6117484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6117916Z hidden_states = self.encoder( 2025-10-10T01:05:25.6118339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6118767Z layer_outputs = layer_module( 2025-10-10T01:05:25.6119157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6119546Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6119979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6120424Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6120856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6121286Z self_outputs = self.self( 2025-10-10T01:05:25.6121734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6122191Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6122352Z 2025-10-10T01:05:25.6122472Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6122863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6123225Z res = mod(**inputs) 2025-10-10T01:05:25.6123647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6124110Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6124548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6124982Z hidden_states = self.encoder( 2025-10-10T01:05:25.6125434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6125894Z layer_outputs = layer_module( 2025-10-10T01:05:25.6126280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6126679Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6127133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6127598Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6128058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6128510Z self_outputs = self.self( 2025-10-10T01:05:25.6128934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6129397Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6129562Z 2025-10-10T01:05:25.6129677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6130080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6130442Z res = mod(**inputs) 2025-10-10T01:05:25.6130852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6131317Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6131989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6132446Z hidden_states = self.encoder( 2025-10-10T01:05:25.6132898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6133352Z layer_outputs = layer_module( 2025-10-10T01:05:25.6133754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6134165Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6134619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6135119Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6135536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6135941Z self_outputs = self.self( 2025-10-10T01:05:25.6136336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6136764Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6136916Z 2025-10-10T01:05:25.6137002Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6137256Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6137503Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6137868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6138190Z res = mod(**inputs) 2025-10-10T01:05:25.6138575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6138992Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6139406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6139811Z hidden_states = self.encoder( 2025-10-10T01:05:25.6140204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6140609Z layer_outputs = layer_module( 2025-10-10T01:05:25.6141016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6141389Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6141802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6142216Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6142654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6143083Z self_outputs = self.self( 2025-10-10T01:05:25.6143500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6143957Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6144133Z 2025-10-10T01:05:25.6144218Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6144484Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6144873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6145221Z res = mod(**inputs) 2025-10-10T01:05:25.6145623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6146080Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6146591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6147093Z hidden_states = self.encoder( 2025-10-10T01:05:25.6147538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6147956Z layer_outputs = layer_module( 2025-10-10T01:05:25.6148316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6148692Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6149114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6149556Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6149979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6150386Z self_outputs = self.self( 2025-10-10T01:05:25.6150783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6151290Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6151796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6152201Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6152342Z 2025-10-10T01:05:25.6152448Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6152809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6153130Z res = mod(**inputs) 2025-10-10T01:05:25.6153508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6153932Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6154343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6154756Z hidden_states = self.encoder( 2025-10-10T01:05:25.6155147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6155582Z layer_outputs = layer_module( 2025-10-10T01:05:25.6155937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6156306Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6156739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6157169Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6157579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6157984Z self_outputs = self.self( 2025-10-10T01:05:25.6158381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6158859Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6159338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6159736Z x = self.pointwise(x) 2025-10-10T01:05:25.6159853Z 2025-10-10T01:05:25.6159957Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6160315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6160633Z res = mod(**inputs) 2025-10-10T01:05:25.6160995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6161401Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6161803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6162193Z hidden_states = self.encoder( 2025-10-10T01:05:25.6162578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6162984Z layer_outputs = layer_module( 2025-10-10T01:05:25.6163334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6163722Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6164131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6164540Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6164956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6165362Z self_outputs = self.self( 2025-10-10T01:05:25.6165762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6166268Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6166477Z 2025-10-10T01:05:25.6166582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6166953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6167288Z res = mod(**inputs) 2025-10-10T01:05:25.6167680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6168117Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6168571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6169011Z hidden_states = self.encoder( 2025-10-10T01:05:25.6169452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6169935Z layer_outputs = layer_module( 2025-10-10T01:05:25.6170310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6170713Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6171162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6171626Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6172087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6172526Z self_outputs = self.self( 2025-10-10T01:05:25.6172974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6173475Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6173665Z 2025-10-10T01:05:25.6173792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6174186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6174532Z res = mod(**inputs) 2025-10-10T01:05:25.6174946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6175397Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6175853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6176293Z hidden_states = self.encoder( 2025-10-10T01:05:25.6176725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6177159Z layer_outputs = layer_module( 2025-10-10T01:05:25.6177541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6177941Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6178374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6178837Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6179252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6179655Z self_outputs = self.self( 2025-10-10T01:05:25.6180044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6180511Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6180711Z 2025-10-10T01:05:25.6180798Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6181029Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6181322Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6181699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6182047Z res = mod(**inputs) 2025-10-10T01:05:25.6182452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6182899Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6183337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6183755Z hidden_states = self.encoder( 2025-10-10T01:05:25.6184176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6184607Z layer_outputs = layer_module( 2025-10-10T01:05:25.6185000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6185408Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6185837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6186370Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6186832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6187286Z self_outputs = self.self( 2025-10-10T01:05:25.6187700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6188180Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6188368Z 2025-10-10T01:05:25.6188491Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6188867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6189200Z res = mod(**inputs) 2025-10-10T01:05:25.6189585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6190011Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6190435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6190847Z hidden_states = self.encoder( 2025-10-10T01:05:25.6191251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6191655Z layer_outputs = layer_module( 2025-10-10T01:05:25.6192010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6192383Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6192802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6193215Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6193637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6194125Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6194586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6195005Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6195147Z 2025-10-10T01:05:25.6195254Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6195620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6195948Z res = mod(**inputs) 2025-10-10T01:05:25.6196353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6196772Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6197177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6197588Z hidden_states = self.encoder( 2025-10-10T01:05:25.6197986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6198392Z layer_outputs = layer_module( 2025-10-10T01:05:25.6198741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6199105Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6199572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6200017Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6200431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6200837Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6201286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6201778Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6202241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6202670Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6202809Z 2025-10-10T01:05:25.6202915Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6203280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6203607Z res = mod(**inputs) 2025-10-10T01:05:25.6203990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6204409Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6204821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6205229Z hidden_states = self.encoder( 2025-10-10T01:05:25.6205631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6206042Z layer_outputs = layer_module( 2025-10-10T01:05:25.6206410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6206810Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6207264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6207703Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6208116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6208533Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6208981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6209474Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6209948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6210393Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6210797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6211142Z return self.act(input) 2025-10-10T01:05:25.6211261Z 2025-10-10T01:05:25.6211366Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6211724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6212045Z res = mod(**inputs) 2025-10-10T01:05:25.6212412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6212822Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6213235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6213644Z hidden_states = self.encoder( 2025-10-10T01:05:25.6214065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6214527Z layer_outputs = layer_module( 2025-10-10T01:05:25.6215337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6215712Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6216157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6216640Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6217079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6217507Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6217972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6218505Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6218994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6219437Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6219593Z 2025-10-10T01:05:25.6219707Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6220098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6220445Z res = mod(**inputs) 2025-10-10T01:05:25.6220843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6221286Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6221727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6222165Z hidden_states = self.encoder( 2025-10-10T01:05:25.6222583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6223019Z layer_outputs = layer_module( 2025-10-10T01:05:25.6223399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6223817Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6224253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6224700Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6225157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6225599Z self_outputs = self.self( 2025-10-10T01:05:25.6226053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6226601Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6226771Z 2025-10-10T01:05:25.6226890Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6227299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6227671Z res = mod(**inputs) 2025-10-10T01:05:25.6228089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6228542Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6228984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6229422Z hidden_states = self.encoder( 2025-10-10T01:05:25.6229886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6230355Z layer_outputs = layer_module( 2025-10-10T01:05:25.6230726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6231129Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6231714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6232177Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6232618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6233051Z self_outputs = self.self( 2025-10-10T01:05:25.6233488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6233938Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6234086Z 2025-10-10T01:05:25.6234202Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6234571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6234899Z res = mod(**inputs) 2025-10-10T01:05:25.6235285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6235705Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6236127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6236527Z hidden_states = self.encoder( 2025-10-10T01:05:25.6236933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6237342Z layer_outputs = layer_module( 2025-10-10T01:05:25.6237703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6238072Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6238481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6238939Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6239353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6239760Z self_outputs = self.self( 2025-10-10T01:05:25.6240153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6240572Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6240731Z 2025-10-10T01:05:25.6240816Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6241038Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6241314Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6241681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6241999Z res = mod(**inputs) 2025-10-10T01:05:25.6242370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6242776Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6243176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6243557Z hidden_states = self.encoder( 2025-10-10T01:05:25.6243939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6244326Z layer_outputs = layer_module( 2025-10-10T01:05:25.6244691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6245063Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6245462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6245867Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6246280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6246669Z self_outputs = self.self( 2025-10-10T01:05:25.6247045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6247470Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6247634Z 2025-10-10T01:05:25.6247717Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6247962Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6248322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6248646Z res = mod(**inputs) 2025-10-10T01:05:25.6249027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6249454Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6249854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6250243Z hidden_states = self.encoder( 2025-10-10T01:05:25.6250632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6251021Z layer_outputs = layer_module( 2025-10-10T01:05:25.6251364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6251726Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6252119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6252521Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6252934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6253320Z self_outputs = self.self( 2025-10-10T01:05:25.6253693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6254160Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6254637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6255048Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6255177Z 2025-10-10T01:05:25.6255285Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6255635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6255945Z res = mod(**inputs) 2025-10-10T01:05:25.6256318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6256730Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6257134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6257524Z hidden_states = self.encoder( 2025-10-10T01:05:25.6257914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6258315Z layer_outputs = layer_module( 2025-10-10T01:05:25.6258699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6259062Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6259457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6259864Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6260269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6260667Z self_outputs = self.self( 2025-10-10T01:05:25.6261051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6261529Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6262013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6262411Z x = self.pointwise(x) 2025-10-10T01:05:25.6262524Z 2025-10-10T01:05:25.6262639Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6263005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6263339Z res = mod(**inputs) 2025-10-10T01:05:25.6263710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6264118Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6264523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6264912Z hidden_states = self.encoder( 2025-10-10T01:05:25.6265306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6265706Z layer_outputs = layer_module( 2025-10-10T01:05:25.6266053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6266512Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6266989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6267452Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6267913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6268308Z self_outputs = self.self( 2025-10-10T01:05:25.6268690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6269197Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6269415Z 2025-10-10T01:05:25.6269521Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6269880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6270201Z res = mod(**inputs) 2025-10-10T01:05:25.6270570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6270981Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6271388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6271786Z hidden_states = self.encoder( 2025-10-10T01:05:25.6272154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6272524Z layer_outputs = layer_module( 2025-10-10T01:05:25.6272894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6273235Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6273619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6274003Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6274385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6274762Z self_outputs = self.self( 2025-10-10T01:05:25.6275128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6275555Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6275716Z 2025-10-10T01:05:25.6275825Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6276170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6276485Z res = mod(**inputs) 2025-10-10T01:05:25.6276851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6277259Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6277657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6278062Z hidden_states = self.encoder( 2025-10-10T01:05:25.6278452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6278858Z layer_outputs = layer_module( 2025-10-10T01:05:25.6279194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6279540Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6279934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6280335Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6280738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6281114Z self_outputs = self.self( 2025-10-10T01:05:25.6281470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6281898Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6282075Z 2025-10-10T01:05:25.6282153Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6282356Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6282574Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6282936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6283242Z res = mod(**inputs) 2025-10-10T01:05:25.6283601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6283990Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6284366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6284742Z hidden_states = self.encoder( 2025-10-10T01:05:25.6285163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6285548Z layer_outputs = layer_module( 2025-10-10T01:05:25.6285882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6286291Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6286688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6287086Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6287486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6287869Z self_outputs = self.self( 2025-10-10T01:05:25.6288247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6288675Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6288834Z 2025-10-10T01:05:25.6288943Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6289293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6289603Z res = mod(**inputs) 2025-10-10T01:05:25.6289971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6290369Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6290767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6291150Z hidden_states = self.encoder( 2025-10-10T01:05:25.6291532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6291924Z layer_outputs = layer_module( 2025-10-10T01:05:25.6292262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6292613Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6293005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6293410Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6293807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6294273Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6294720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6295124Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6295273Z 2025-10-10T01:05:25.6295380Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6295734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6296062Z res = mod(**inputs) 2025-10-10T01:05:25.6296466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6296880Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6297327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6297770Z hidden_states = self.encoder( 2025-10-10T01:05:25.6298177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6298633Z layer_outputs = layer_module( 2025-10-10T01:05:25.6298979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6299343Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6299735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6300186Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6300583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6300975Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6301405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6301894Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6302350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6302761Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6302911Z 2025-10-10T01:05:25.6303019Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6303387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6303722Z res = mod(**inputs) 2025-10-10T01:05:25.6304096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6304510Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6304926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6305331Z hidden_states = self.encoder( 2025-10-10T01:05:25.6305733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6306152Z layer_outputs = layer_module( 2025-10-10T01:05:25.6306599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6306995Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6307453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6307914Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6308346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6308804Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6309274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6309763Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6310220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6310667Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6311084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6311446Z return self.act(input) 2025-10-10T01:05:25.6311565Z 2025-10-10T01:05:25.6311686Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6312057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6312395Z res = mod(**inputs) 2025-10-10T01:05:25.6312788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6313289Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6313710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6314120Z hidden_states = self.encoder( 2025-10-10T01:05:25.6314529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6314985Z layer_outputs = layer_module( 2025-10-10T01:05:25.6315339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6315711Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6316119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6316534Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6316970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6317392Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6317850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6318379Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6318876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6319296Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6319441Z 2025-10-10T01:05:25.6319557Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6319917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6320246Z res = mod(**inputs) 2025-10-10T01:05:25.6320630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6321045Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6321316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6321398Z hidden_states = self.encoder( 2025-10-10T01:05:25.6321673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6321748Z layer_outputs = layer_module( 2025-10-10T01:05:25.6321984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6322085Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6322362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6322445Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6322725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6322801Z self_outputs = self.self( 2025-10-10T01:05:25.6323072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6323198Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6323202Z 2025-10-10T01:05:25.6323312Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6323532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6323606Z res = mod(**inputs) 2025-10-10T01:05:25.6323891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6323996Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6324264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6324346Z hidden_states = self.encoder( 2025-10-10T01:05:25.6324616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6324736Z layer_outputs = layer_module( 2025-10-10T01:05:25.6324967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6325047Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6325325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6325408Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6325683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6325757Z self_outputs = self.self( 2025-10-10T01:05:25.6326045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6326143Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6326147Z 2025-10-10T01:05:25.6326268Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6326490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6326561Z res = mod(**inputs) 2025-10-10T01:05:25.6326857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6326946Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6327232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6327318Z hidden_states = self.encoder( 2025-10-10T01:05:25.6327603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6327686Z layer_outputs = layer_module( 2025-10-10T01:05:25.6327930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6328017Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6328311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6328419Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6328710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6328788Z self_outputs = self.self( 2025-10-10T01:05:25.6329079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6329175Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6329179Z 2025-10-10T01:05:25.6329264Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6329355Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6329492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6329706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6329774Z res = mod(**inputs) 2025-10-10T01:05:25.6330043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6330146Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6330409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6330487Z hidden_states = self.encoder( 2025-10-10T01:05:25.6330748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6330820Z layer_outputs = layer_module( 2025-10-10T01:05:25.6331067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6331162Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6331548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6331644Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6331923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6331996Z self_outputs = self.self( 2025-10-10T01:05:25.6332275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6332385Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6332389Z 2025-10-10T01:05:25.6332470Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6332582Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6332784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6332852Z res = mod(**inputs) 2025-10-10T01:05:25.6333127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6333211Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6333479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6333553Z hidden_states = self.encoder( 2025-10-10T01:05:25.6333817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6333898Z layer_outputs = layer_module( 2025-10-10T01:05:25.6334118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6334207Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6334473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6334562Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6334871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6334943Z self_outputs = self.self( 2025-10-10T01:05:25.6335213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6335377Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6335650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6335729Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6335733Z 2025-10-10T01:05:25.6335873Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6336071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6336138Z res = mod(**inputs) 2025-10-10T01:05:25.6336408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6336491Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6336761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6336835Z hidden_states = self.encoder( 2025-10-10T01:05:25.6337105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6337186Z layer_outputs = layer_module( 2025-10-10T01:05:25.6337450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6337562Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6337821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6337903Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6338174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6338244Z self_outputs = self.self( 2025-10-10T01:05:25.6338513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6338671Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6338942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6339014Z x = self.pointwise(x) 2025-10-10T01:05:25.6339017Z 2025-10-10T01:05:25.6339120Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6339320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6339388Z res = mod(**inputs) 2025-10-10T01:05:25.6339655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6339737Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6340000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6340078Z hidden_states = self.encoder( 2025-10-10T01:05:25.6340339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6340421Z layer_outputs = layer_module( 2025-10-10T01:05:25.6340641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6340724Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6340986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6341085Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6341361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6341432Z self_outputs = self.self( 2025-10-10T01:05:25.6341699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6341852Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6341857Z 2025-10-10T01:05:25.6341981Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6342183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6342249Z res = mod(**inputs) 2025-10-10T01:05:25.6342522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6342604Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6342874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6342945Z hidden_states = self.encoder( 2025-10-10T01:05:25.6343207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6343285Z layer_outputs = layer_module( 2025-10-10T01:05:25.6343527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6343634Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6343895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6343977Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6344244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6344314Z self_outputs = self.self( 2025-10-10T01:05:25.6344583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6344706Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6344710Z 2025-10-10T01:05:25.6344823Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6345025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6345096Z res = mod(**inputs) 2025-10-10T01:05:25.6345378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6345466Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6345757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6345835Z hidden_states = self.encoder( 2025-10-10T01:05:25.6346119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6346266Z layer_outputs = layer_module( 2025-10-10T01:05:25.6346514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6346607Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6346911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6347009Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6347309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6347405Z self_outputs = self.self( 2025-10-10T01:05:25.6347678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6347807Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6347811Z 2025-10-10T01:05:25.6347900Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6347979Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6348085Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6348310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6348379Z res = mod(**inputs) 2025-10-10T01:05:25.6348663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6348745Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6349002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6349079Z hidden_states = self.encoder( 2025-10-10T01:05:25.6349333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6349409Z layer_outputs = layer_module( 2025-10-10T01:05:25.6349622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6349704Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6350003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6350084Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6350345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6350416Z self_outputs = self.self( 2025-10-10T01:05:25.6350678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6350789Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6350792Z 2025-10-10T01:05:25.6350892Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6351093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6351170Z res = mod(**inputs) 2025-10-10T01:05:25.6351439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6351518Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6351780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6351851Z hidden_states = self.encoder( 2025-10-10T01:05:25.6352107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6352183Z layer_outputs = layer_module( 2025-10-10T01:05:25.6352394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6352478Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6352737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6352817Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6353079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6353236Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6353498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6353580Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6353583Z 2025-10-10T01:05:25.6353691Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6353878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6353942Z res = mod(**inputs) 2025-10-10T01:05:25.6354221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6354302Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6354564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6354635Z hidden_states = self.encoder( 2025-10-10T01:05:25.6354890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6354967Z layer_outputs = layer_module( 2025-10-10T01:05:25.6355177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6355257Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6355513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6355595Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6355889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6355969Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6356263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6356383Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6356645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6356728Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6356732Z 2025-10-10T01:05:25.6356836Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6357038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6357102Z res = mod(**inputs) 2025-10-10T01:05:25.6357376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6357458Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6357729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6357802Z hidden_states = self.encoder( 2025-10-10T01:05:25.6358064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6358143Z layer_outputs = layer_module( 2025-10-10T01:05:25.6358361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6358448Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6358720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6358802Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6359058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6359156Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6359458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6359580Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6359844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6359965Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6360180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6360317Z return self.act(input) 2025-10-10T01:05:25.6360321Z 2025-10-10T01:05:25.6360426Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6360629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6360697Z res = mod(**inputs) 2025-10-10T01:05:25.6360965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6361056Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6361319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6361399Z hidden_states = self.encoder( 2025-10-10T01:05:25.6361661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6361732Z layer_outputs = layer_module( 2025-10-10T01:05:25.6361997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6362077Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6362347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6362431Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6362694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6362772Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6363067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6363207Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6363475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6363566Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6363569Z 2025-10-10T01:05:25.6363673Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6363867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6363944Z res = mod(**inputs) 2025-10-10T01:05:25.6364210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6364299Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6364563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6364641Z hidden_states = self.encoder( 2025-10-10T01:05:25.6364909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6364981Z layer_outputs = layer_module( 2025-10-10T01:05:25.6365208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6365328Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6365600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6365681Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6365946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6366026Z self_outputs = self.self( 2025-10-10T01:05:25.6366292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6366410Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6366416Z 2025-10-10T01:05:25.6366520Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6366723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6366791Z res = mod(**inputs) 2025-10-10T01:05:25.6367056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6367147Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6367411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6367490Z hidden_states = self.encoder( 2025-10-10T01:05:25.6367756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6367827Z layer_outputs = layer_module( 2025-10-10T01:05:25.6368093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6368171Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6368501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6368586Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6368860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6368932Z self_outputs = self.self( 2025-10-10T01:05:25.6369198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6369287Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6369291Z 2025-10-10T01:05:25.6369395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6369603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6369668Z res = mod(**inputs) 2025-10-10T01:05:25.6369933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6370023Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6370286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6370365Z hidden_states = self.encoder( 2025-10-10T01:05:25.6370628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6370697Z layer_outputs = layer_module( 2025-10-10T01:05:25.6370924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6371005Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6371277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6371359Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6371656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6371727Z self_outputs = self.self( 2025-10-10T01:05:25.6371998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6372095Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6372099Z 2025-10-10T01:05:25.6372178Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6372262Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6372364Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6372576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6372649Z res = mod(**inputs) 2025-10-10T01:05:25.6372905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6372993Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6373252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6373329Z hidden_states = self.encoder( 2025-10-10T01:05:25.6373584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6373651Z layer_outputs = layer_module( 2025-10-10T01:05:25.6373872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6373992Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6374258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6374338Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6374596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6374673Z self_outputs = self.self( 2025-10-10T01:05:25.6374931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6375040Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6375044Z 2025-10-10T01:05:25.6375122Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6375222Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6375425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6375492Z res = mod(**inputs) 2025-10-10T01:05:25.6375759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6375839Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6376109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6376182Z hidden_states = self.encoder( 2025-10-10T01:05:25.6376449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6376528Z layer_outputs = layer_module( 2025-10-10T01:05:25.6376752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6376839Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6377115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6377197Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6377481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6377575Z self_outputs = self.self( 2025-10-10T01:05:25.6377852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6378025Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6378304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6378382Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6378385Z 2025-10-10T01:05:25.6378507Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6378713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6378780Z res = mod(**inputs) 2025-10-10T01:05:25.6379050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6379132Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6379396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6379475Z hidden_states = self.encoder( 2025-10-10T01:05:25.6379741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6379820Z layer_outputs = layer_module( 2025-10-10T01:05:25.6380048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6381153Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6381447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6381530Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6381812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6381885Z self_outputs = self.self( 2025-10-10T01:05:25.6382163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6382324Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6382593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6382679Z x = self.pointwise(x) 2025-10-10T01:05:25.6382684Z 2025-10-10T01:05:25.6382792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6382999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6383067Z res = mod(**inputs) 2025-10-10T01:05:25.6383345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6383429Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6383699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6383780Z hidden_states = self.encoder( 2025-10-10T01:05:25.6384084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6384171Z layer_outputs = layer_module( 2025-10-10T01:05:25.6384414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6384499Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6384800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6384911Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6385210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6385285Z self_outputs = self.self( 2025-10-10T01:05:25.6385586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6385750Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6385755Z 2025-10-10T01:05:25.6385886Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6386111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6386251Z res = mod(**inputs) 2025-10-10T01:05:25.6386566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6386657Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6386942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6387028Z hidden_states = self.encoder( 2025-10-10T01:05:25.6387325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6387410Z layer_outputs = layer_module( 2025-10-10T01:05:25.6387703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6387834Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6388095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6388174Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6388448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6388519Z self_outputs = self.self( 2025-10-10T01:05:25.6388861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6388994Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6388998Z 2025-10-10T01:05:25.6389108Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6389328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6389404Z res = mod(**inputs) 2025-10-10T01:05:25.6389696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6389784Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6390077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6390155Z hidden_states = self.encoder( 2025-10-10T01:05:25.6390450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6390535Z layer_outputs = layer_module( 2025-10-10T01:05:25.6390816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6390910Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6391210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6391300Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6391602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6391700Z self_outputs = self.self( 2025-10-10T01:05:25.6392002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6392141Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6392145Z 2025-10-10T01:05:25.6392241Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6392330Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6392444Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6392686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6392763Z res = mod(**inputs) 2025-10-10T01:05:25.6393058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6393146Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6393430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6393514Z hidden_states = self.encoder( 2025-10-10T01:05:25.6393813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6393899Z layer_outputs = layer_module( 2025-10-10T01:05:25.6394138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6394225Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6394559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6394642Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6394908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6394978Z self_outputs = self.self( 2025-10-10T01:05:25.6395238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6395356Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6395360Z 2025-10-10T01:05:25.6395464Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6395670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6395739Z res = mod(**inputs) 2025-10-10T01:05:25.6396017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6396101Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6396368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6396450Z hidden_states = self.encoder( 2025-10-10T01:05:25.6396718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6396796Z layer_outputs = layer_module( 2025-10-10T01:05:25.6397020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6397098Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6397373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6397470Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6397739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6397869Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6398157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6398240Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6398244Z 2025-10-10T01:05:25.6398347Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6398550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6398616Z res = mod(**inputs) 2025-10-10T01:05:25.6398886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6398989Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6399257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6399335Z hidden_states = self.encoder( 2025-10-10T01:05:25.6399602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6399679Z layer_outputs = layer_module( 2025-10-10T01:05:25.6399900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6399982Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6400243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6400327Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6400620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6400719Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6401022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6401145Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6401407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6401495Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6401499Z 2025-10-10T01:05:25.6401604Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6401807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6401872Z res = mod(**inputs) 2025-10-10T01:05:25.6402141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6402222Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6402483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6402562Z hidden_states = self.encoder( 2025-10-10T01:05:25.6402825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6402902Z layer_outputs = layer_module( 2025-10-10T01:05:25.6403120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6403199Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6403466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6403555Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6403815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6403893Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6404217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6404340Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6404610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6404733Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6404951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6405030Z return self.act(input) 2025-10-10T01:05:25.6405036Z 2025-10-10T01:05:25.6405163Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6405376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6405449Z res = mod(**inputs) 2025-10-10T01:05:25.6405711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6405799Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6406059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6406138Z hidden_states = self.encoder( 2025-10-10T01:05:25.6406397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6406469Z layer_outputs = layer_module( 2025-10-10T01:05:25.6406711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6406807Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6407074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6407159Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6407413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6407497Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6407789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6407930Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6408196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6408287Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6408291Z 2025-10-10T01:05:25.6408395Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6408590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6408666Z res = mod(**inputs) 2025-10-10T01:05:25.6408928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6409016Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6409277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6409349Z hidden_states = self.encoder( 2025-10-10T01:05:25.6409618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6409694Z layer_outputs = layer_module( 2025-10-10T01:05:25.6409918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6409997Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6410282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6410362Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6410621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6410701Z self_outputs = self.self( 2025-10-10T01:05:25.6410962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6411060Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6411065Z 2025-10-10T01:05:25.6411189Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6411383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6411460Z res = mod(**inputs) 2025-10-10T01:05:25.6411724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6411816Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6412081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6412161Z hidden_states = self.encoder( 2025-10-10T01:05:25.6412426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6412499Z layer_outputs = layer_module( 2025-10-10T01:05:25.6412746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6412842Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6413112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6413196Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6413461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6413542Z self_outputs = self.self( 2025-10-10T01:05:25.6413804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6413894Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6413898Z 2025-10-10T01:05:25.6414000Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6414205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6414273Z res = mod(**inputs) 2025-10-10T01:05:25.6414539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6414633Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6414902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6414983Z hidden_states = self.encoder( 2025-10-10T01:05:25.6415252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6415323Z layer_outputs = layer_module( 2025-10-10T01:05:25.6415554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6415633Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6415914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6415998Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6416266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6416375Z self_outputs = self.self( 2025-10-10T01:05:25.6416636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6416732Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6416736Z 2025-10-10T01:05:25.6416815Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6416901Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6417005Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6417227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6417305Z res = mod(**inputs) 2025-10-10T01:05:25.6417576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6417666Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6417940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6418012Z hidden_states = self.encoder( 2025-10-10T01:05:25.6418296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6418367Z layer_outputs = layer_module( 2025-10-10T01:05:25.6418600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6418682Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6418997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6419089Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6419355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6419437Z self_outputs = self.self( 2025-10-10T01:05:25.6419706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6419820Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6419824Z 2025-10-10T01:05:25.6419906Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6420013Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6420222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6420293Z res = mod(**inputs) 2025-10-10T01:05:25.6420573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6420661Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6420948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6421033Z hidden_states = self.encoder( 2025-10-10T01:05:25.6421321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6421406Z layer_outputs = layer_module( 2025-10-10T01:05:25.6421642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6421733Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6422021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6422106Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6422382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6422476Z self_outputs = self.self( 2025-10-10T01:05:25.6422753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6422917Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6423196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6423289Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6423293Z 2025-10-10T01:05:25.6423405Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6423658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6423731Z res = mod(**inputs) 2025-10-10T01:05:25.6424024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6424114Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6424400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6424486Z hidden_states = self.encoder( 2025-10-10T01:05:25.6424772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6424857Z layer_outputs = layer_module( 2025-10-10T01:05:25.6425101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6425208Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6425519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6425607Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6425899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6425975Z self_outputs = self.self( 2025-10-10T01:05:25.6426344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6426535Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6426831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6426916Z x = self.pointwise(x) 2025-10-10T01:05:25.6426924Z 2025-10-10T01:05:25.6427040Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6427261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6427332Z res = mod(**inputs) 2025-10-10T01:05:25.6427621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6427718Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6427986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6428071Z hidden_states = self.encoder( 2025-10-10T01:05:25.6428339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6428412Z layer_outputs = layer_module( 2025-10-10T01:05:25.6428647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6428730Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6429008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6429117Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6429394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6429466Z self_outputs = self.self( 2025-10-10T01:05:25.6429738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6429900Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6429905Z 2025-10-10T01:05:25.6430010Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6430239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6430307Z res = mod(**inputs) 2025-10-10T01:05:25.6430574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6430665Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6430936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6431017Z hidden_states = self.encoder( 2025-10-10T01:05:25.6431291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6431372Z layer_outputs = layer_module( 2025-10-10T01:05:25.6431819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6431908Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6432265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6432350Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6432627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6432699Z self_outputs = self.self( 2025-10-10T01:05:25.6432972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6433104Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6433108Z 2025-10-10T01:05:25.6433223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6433428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6433493Z res = mod(**inputs) 2025-10-10T01:05:25.6433769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6433852Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6434115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6434194Z hidden_states = self.encoder( 2025-10-10T01:05:25.6434457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6434536Z layer_outputs = layer_module( 2025-10-10T01:05:25.6434756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6434833Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6435107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6435188Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6435458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6435556Z self_outputs = self.self( 2025-10-10T01:05:25.6435823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6435949Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6435952Z 2025-10-10T01:05:25.6436032Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6436119Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6436223Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6436424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6436492Z res = mod(**inputs) 2025-10-10T01:05:25.6436791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6436883Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6437146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6437227Z hidden_states = self.encoder( 2025-10-10T01:05:25.6437488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6437557Z layer_outputs = layer_module( 2025-10-10T01:05:25.6437782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6437860Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6438148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6438249Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6438516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6438589Z self_outputs = self.self( 2025-10-10T01:05:25.6438852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6438972Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6438975Z 2025-10-10T01:05:25.6439078Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6439280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6439346Z res = mod(**inputs) 2025-10-10T01:05:25.6439610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6439702Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6439964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6440043Z hidden_states = self.encoder( 2025-10-10T01:05:25.6440307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6440384Z layer_outputs = layer_module( 2025-10-10T01:05:25.6440602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6440680Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6440946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6441026Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6441299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6441429Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6441701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6441815Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6441818Z 2025-10-10T01:05:25.6441921Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6442124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6442191Z res = mod(**inputs) 2025-10-10T01:05:25.6442461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6442543Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6442825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6442906Z hidden_states = self.encoder( 2025-10-10T01:05:25.6443170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6443248Z layer_outputs = layer_module( 2025-10-10T01:05:25.6443467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6443546Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6443819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6443906Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6444195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6444293Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6444595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6444719Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6444982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6445072Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6445075Z 2025-10-10T01:05:25.6445180Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6445384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6445449Z res = mod(**inputs) 2025-10-10T01:05:25.6445715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6445806Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6446067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6446148Z hidden_states = self.encoder( 2025-10-10T01:05:25.6446411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6446488Z layer_outputs = layer_module( 2025-10-10T01:05:25.6446710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6446788Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6447057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6447141Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6447406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6447484Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6447779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6447930Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6448203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6448320Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6448529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6448606Z return self.act(input) 2025-10-10T01:05:25.6448609Z 2025-10-10T01:05:25.6448732Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6448923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6448994Z res = mod(**inputs) 2025-10-10T01:05:25.6467718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6468049Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6468401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6468487Z hidden_states = self.encoder( 2025-10-10T01:05:25.6468775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6468864Z layer_outputs = layer_module( 2025-10-10T01:05:25.6469218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6469367Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6469646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6469753Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6470024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6470111Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6470431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6470573Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6470854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6470950Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6470957Z 2025-10-10T01:05:25.6471085Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6471299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6471373Z res = mod(**inputs) 2025-10-10T01:05:25.6471659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6471753Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6472035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6472113Z hidden_states = self.encoder( 2025-10-10T01:05:25.6472384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6472466Z layer_outputs = layer_module( 2025-10-10T01:05:25.6472703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6472793Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6473065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6473189Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6473464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6473543Z self_outputs = self.self( 2025-10-10T01:05:25.6473823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6473921Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6473926Z 2025-10-10T01:05:25.6474046Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6474290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6474363Z res = mod(**inputs) 2025-10-10T01:05:25.6474644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6474731Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6475014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6475090Z hidden_states = self.encoder( 2025-10-10T01:05:25.6475364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6475445Z layer_outputs = layer_module( 2025-10-10T01:05:25.6475675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6475801Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6476074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6476169Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6476457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6476533Z self_outputs = self.self( 2025-10-10T01:05:25.6476813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6476902Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6476906Z 2025-10-10T01:05:25.6477026Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6477235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6477310Z res = mod(**inputs) 2025-10-10T01:05:25.6477591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6477676Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6477954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6478029Z hidden_states = self.encoder( 2025-10-10T01:05:25.6478307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6478380Z layer_outputs = layer_module( 2025-10-10T01:05:25.6478607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6478697Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6478968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6479061Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6479329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6479426Z self_outputs = self.self( 2025-10-10T01:05:25.6479705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6479801Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6479805Z 2025-10-10T01:05:25.6479900Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6479982Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6480090Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6480316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6480421Z res = mod(**inputs) 2025-10-10T01:05:25.6480695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6480778Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6481051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6481122Z hidden_states = self.encoder( 2025-10-10T01:05:25.6481386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6481466Z layer_outputs = layer_module( 2025-10-10T01:05:25.6481688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6481775Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6482054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6482155Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6482426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6482500Z self_outputs = self.self( 2025-10-10T01:05:25.6482772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6482879Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6482883Z 2025-10-10T01:05:25.6482974Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6483079Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6483278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6483358Z res = mod(**inputs) 2025-10-10T01:05:25.6483628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6483719Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6483984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6484060Z hidden_states = self.encoder( 2025-10-10T01:05:25.6484332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6484408Z layer_outputs = layer_module( 2025-10-10T01:05:25.6484646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6484726Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6484999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6485096Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6485368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6485448Z self_outputs = self.self( 2025-10-10T01:05:25.6485747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6485934Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6486278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6486365Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6486369Z 2025-10-10T01:05:25.6486492Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6486725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6486807Z res = mod(**inputs) 2025-10-10T01:05:25.6487093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6487182Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6487482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6487561Z hidden_states = self.encoder( 2025-10-10T01:05:25.6487854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6487931Z layer_outputs = layer_module( 2025-10-10T01:05:25.6488181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6488274Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6488586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6488677Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6488949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6489029Z self_outputs = self.self( 2025-10-10T01:05:25.6489300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6489467Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6489747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6489822Z x = self.pointwise(x) 2025-10-10T01:05:25.6489826Z 2025-10-10T01:05:25.6489944Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6490159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6490232Z res = mod(**inputs) 2025-10-10T01:05:25.6490495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6490578Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6490849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6490924Z hidden_states = self.encoder( 2025-10-10T01:05:25.6491193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6491265Z layer_outputs = layer_module( 2025-10-10T01:05:25.6491489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6491582Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6491853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6491945Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6492232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6492313Z self_outputs = self.self( 2025-10-10T01:05:25.6492582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6492748Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6492752Z 2025-10-10T01:05:25.6492865Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6493075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6493152Z res = mod(**inputs) 2025-10-10T01:05:25.6493416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6493497Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6493771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6493846Z hidden_states = self.encoder( 2025-10-10T01:05:25.6494122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6494197Z layer_outputs = layer_module( 2025-10-10T01:05:25.6494430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6494511Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6494820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6494914Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6495187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6495272Z self_outputs = self.self( 2025-10-10T01:05:25.6495546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6495672Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6495676Z 2025-10-10T01:05:25.6495792Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6495995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6496072Z res = mod(**inputs) 2025-10-10T01:05:25.6496349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6496445Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6496717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6496793Z hidden_states = self.encoder( 2025-10-10T01:05:25.6497083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6497161Z layer_outputs = layer_module( 2025-10-10T01:05:25.6497408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6497493Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6497780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6497878Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6498150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6498234Z self_outputs = self.self( 2025-10-10T01:05:25.6498525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6498659Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6498670Z 2025-10-10T01:05:25.6498755Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6498839Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6498956Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6499159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6499237Z res = mod(**inputs) 2025-10-10T01:05:25.6499539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6499626Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6499909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6499987Z hidden_states = self.encoder( 2025-10-10T01:05:25.6500270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6500344Z layer_outputs = layer_module( 2025-10-10T01:05:25.6500574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6500666Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6500968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6501106Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6501395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6501470Z self_outputs = self.self( 2025-10-10T01:05:25.6501773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6501899Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6501903Z 2025-10-10T01:05:25.6502028Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6502243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6502323Z res = mod(**inputs) 2025-10-10T01:05:25.6502625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6502717Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6503020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6503099Z hidden_states = self.encoder( 2025-10-10T01:05:25.6503399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6503474Z layer_outputs = layer_module( 2025-10-10T01:05:25.6503712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6503803Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6504106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6504206Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6504511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6504663Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6504953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6505077Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6505082Z 2025-10-10T01:05:25.6505205Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6505417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6505499Z res = mod(**inputs) 2025-10-10T01:05:25.6505809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6505899Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6506335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6506429Z hidden_states = self.encoder( 2025-10-10T01:05:25.6506733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6506814Z layer_outputs = layer_module( 2025-10-10T01:05:25.6507066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6507153Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6507459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6507571Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6507839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6507966Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6508272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6508400Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6508679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6508766Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6508770Z 2025-10-10T01:05:25.6508884Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6509086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6509164Z res = mod(**inputs) 2025-10-10T01:05:25.6509432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6509520Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6509797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6509873Z hidden_states = self.encoder( 2025-10-10T01:05:25.6510148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6510221Z layer_outputs = layer_module( 2025-10-10T01:05:25.6510446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6510533Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6510817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6510918Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6511199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6511293Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6511609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6511754Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6512031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6512148Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6512376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6512451Z return self.act(input) 2025-10-10T01:05:25.6512455Z 2025-10-10T01:05:25.6512563Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6512793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6512863Z res = mod(**inputs) 2025-10-10T01:05:25.6513142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6513228Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6513509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6513585Z hidden_states = self.encoder( 2025-10-10T01:05:25.6513858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6513940Z layer_outputs = layer_module( 2025-10-10T01:05:25.6514168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6514295Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6514567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6514652Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6514926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6515006Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6515315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6515454Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6515730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6515814Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6515821Z 2025-10-10T01:05:25.6515937Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6516141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6516220Z res = mod(**inputs) 2025-10-10T01:05:25.6516512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6516601Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6516900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6516976Z hidden_states = self.encoder( 2025-10-10T01:05:25.6517277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6517361Z layer_outputs = layer_module( 2025-10-10T01:05:25.6517608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6517701Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6517991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6518109Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6518396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6518476Z self_outputs = self.self( 2025-10-10T01:05:25.6518766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6518864Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6518868Z 2025-10-10T01:05:25.6518983Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6519215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6519291Z res = mod(**inputs) 2025-10-10T01:05:25.6519583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6519673Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6519963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6520040Z hidden_states = self.encoder( 2025-10-10T01:05:25.6520331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6520408Z layer_outputs = layer_module( 2025-10-10T01:05:25.6520648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6520738Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6521065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6521159Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6521445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6521524Z self_outputs = self.self( 2025-10-10T01:05:25.6521817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6521906Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6521910Z 2025-10-10T01:05:25.6522030Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6522240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6522310Z res = mod(**inputs) 2025-10-10T01:05:25.6522612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6522698Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6522990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6523069Z hidden_states = self.encoder( 2025-10-10T01:05:25.6523360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6523437Z layer_outputs = layer_module( 2025-10-10T01:05:25.6523683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6523777Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6524083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6524178Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6524463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6524559Z self_outputs = self.self( 2025-10-10T01:05:25.6524848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6524945Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6524949Z 2025-10-10T01:05:25.6525043Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6525130Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6525248Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6525462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6525532Z res = mod(**inputs) 2025-10-10T01:05:25.6525849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6525938Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6526231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6526310Z hidden_states = self.encoder( 2025-10-10T01:05:25.6526597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6526681Z layer_outputs = layer_module( 2025-10-10T01:05:25.6526920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6527012Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6527302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6527434Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6527725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6527800Z self_outputs = self.self( 2025-10-10T01:05:25.6528094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6528205Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6528209Z 2025-10-10T01:05:25.6528301Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6528414Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6528625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6528704Z res = mod(**inputs) 2025-10-10T01:05:25.6528993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6529091Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6529372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6529452Z hidden_states = self.encoder( 2025-10-10T01:05:25.6529741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6529817Z layer_outputs = layer_module( 2025-10-10T01:05:25.6530062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6530148Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6530437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6530525Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6530812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6530894Z self_outputs = self.self( 2025-10-10T01:05:25.6531178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6531378Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6531883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6531975Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6531979Z 2025-10-10T01:05:25.6532100Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6532314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6532459Z res = mod(**inputs) 2025-10-10T01:05:25.6532752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6532846Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6533132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6533211Z hidden_states = self.encoder( 2025-10-10T01:05:25.6533510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6533581Z layer_outputs = layer_module( 2025-10-10T01:05:25.6533815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6533897Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6534204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6534328Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6534620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6534706Z self_outputs = self.self( 2025-10-10T01:05:25.6534997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6535175Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6535465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6535541Z x = self.pointwise(x) 2025-10-10T01:05:25.6535545Z 2025-10-10T01:05:25.6535665Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6535881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6535960Z res = mod(**inputs) 2025-10-10T01:05:25.6536253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6536342Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6536640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6536717Z hidden_states = self.encoder( 2025-10-10T01:05:25.6537013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6537089Z layer_outputs = layer_module( 2025-10-10T01:05:25.6537331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6537423Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6537716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6537813Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6538104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6538215Z self_outputs = self.self( 2025-10-10T01:05:25.6538483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6538639Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6538643Z 2025-10-10T01:05:25.6538755Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6538955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6539028Z res = mod(**inputs) 2025-10-10T01:05:25.6539319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6539403Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6539678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6539752Z hidden_states = self.encoder( 2025-10-10T01:05:25.6540031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6540103Z layer_outputs = layer_module( 2025-10-10T01:05:25.6540334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6540413Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6540703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6540811Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6541095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6541179Z self_outputs = self.self( 2025-10-10T01:05:25.6541464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6541595Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6541606Z 2025-10-10T01:05:25.6541717Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6541929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6542004Z res = mod(**inputs) 2025-10-10T01:05:25.6542296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6542387Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6542656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6542736Z hidden_states = self.encoder( 2025-10-10T01:05:25.6543032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6543109Z layer_outputs = layer_module( 2025-10-10T01:05:25.6543355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6543437Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6543735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6543829Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6544121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6544203Z self_outputs = self.self( 2025-10-10T01:05:25.6544490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6544657Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6544661Z 2025-10-10T01:05:25.6544748Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6544834Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6544952Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6545160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6545238Z res = mod(**inputs) 2025-10-10T01:05:25.6545559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6545650Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6545944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6546025Z hidden_states = self.encoder( 2025-10-10T01:05:25.6546385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6546470Z layer_outputs = layer_module( 2025-10-10T01:05:25.6546719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6546814Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6547124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6547221Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6547577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6547675Z self_outputs = self.self( 2025-10-10T01:05:25.6547942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6548063Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6548067Z 2025-10-10T01:05:25.6548179Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6548382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6548459Z res = mod(**inputs) 2025-10-10T01:05:25.6548730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6548815Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6549099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6549173Z hidden_states = self.encoder( 2025-10-10T01:05:25.6549455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6549531Z layer_outputs = layer_module( 2025-10-10T01:05:25.6549766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6549844Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6550110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6550203Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6550475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6550619Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6550889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6550995Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6551006Z 2025-10-10T01:05:25.6551110Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6551308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6551381Z res = mod(**inputs) 2025-10-10T01:05:25.6551649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6551736Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6552021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6552098Z hidden_states = self.encoder( 2025-10-10T01:05:25.6552376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6552451Z layer_outputs = layer_module( 2025-10-10T01:05:25.6552687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6552765Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6553035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6553129Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6553398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6553486Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6553845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6553977Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6554238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6554322Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6554326Z 2025-10-10T01:05:25.6554436Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6554630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6554702Z res = mod(**inputs) 2025-10-10T01:05:25.6554965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6555046Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6555318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6555390Z hidden_states = self.encoder( 2025-10-10T01:05:25.6555657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6555731Z layer_outputs = layer_module( 2025-10-10T01:05:25.6555953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6556032Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6556293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6556385Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6556648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6556735Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6557037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6557181Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6557457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6557572Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6557798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6557872Z return self.act(input) 2025-10-10T01:05:25.6557876Z 2025-10-10T01:05:25.6557988Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6558220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6558288Z res = mod(**inputs) 2025-10-10T01:05:25.6558559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6558639Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6558910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6558982Z hidden_states = self.encoder( 2025-10-10T01:05:25.6559244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6559323Z layer_outputs = layer_module( 2025-10-10T01:05:25.6559539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6559625Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6559927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6560015Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6560287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6560367Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6560680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6560817Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6561096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6561180Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6561184Z 2025-10-10T01:05:25.6561294Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6561501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6561568Z res = mod(**inputs) 2025-10-10T01:05:25.6561846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6561931Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6562209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6562284Z hidden_states = self.encoder( 2025-10-10T01:05:25.6562556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6562637Z layer_outputs = layer_module( 2025-10-10T01:05:25.6562866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6562955Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6563227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6563338Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6563617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6563690Z self_outputs = self.self( 2025-10-10T01:05:25.6563967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6564061Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6564065Z 2025-10-10T01:05:25.6564177Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6564406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6564477Z res = mod(**inputs) 2025-10-10T01:05:25.6564749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6564830Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6565101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6565172Z hidden_states = self.encoder( 2025-10-10T01:05:25.6565433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6565510Z layer_outputs = layer_module( 2025-10-10T01:05:25.6565728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6565811Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6566110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6566200Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6566490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6566569Z self_outputs = self.self( 2025-10-10T01:05:25.6566861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6566950Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6566954Z 2025-10-10T01:05:25.6567072Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6567284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6567353Z res = mod(**inputs) 2025-10-10T01:05:25.6567651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6567741Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6568034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6568123Z hidden_states = self.encoder( 2025-10-10T01:05:25.6568387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6568464Z layer_outputs = layer_module( 2025-10-10T01:05:25.6568681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6568766Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6569029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6569121Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6569385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6569458Z self_outputs = self.self( 2025-10-10T01:05:25.6569754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6569850Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6569854Z 2025-10-10T01:05:25.6569946Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6570029Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6570135Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6570345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6570412Z res = mod(**inputs) 2025-10-10T01:05:25.6570716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6570807Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6571088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6571173Z hidden_states = self.encoder( 2025-10-10T01:05:25.6571459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6571541Z layer_outputs = layer_module( 2025-10-10T01:05:25.6571777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6571867Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6572150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6572278Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6572571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6572646Z self_outputs = self.self( 2025-10-10T01:05:25.6572939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6573054Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6573058Z 2025-10-10T01:05:25.6573142Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6573260Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6573470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6573546Z res = mod(**inputs) 2025-10-10T01:05:25.6573828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6573926Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6574208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6574284Z hidden_states = self.encoder( 2025-10-10T01:05:25.6574577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6574652Z layer_outputs = layer_module( 2025-10-10T01:05:25.6574893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6574978Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6575260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6575354Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6575642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6575725Z self_outputs = self.self( 2025-10-10T01:05:25.6576009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6576204Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6576499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6576584Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6576588Z 2025-10-10T01:05:25.6576707Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6576918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6576995Z res = mod(**inputs) 2025-10-10T01:05:25.6577318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6577408Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6577705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6577784Z hidden_states = self.encoder( 2025-10-10T01:05:25.6578077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6578155Z layer_outputs = layer_module( 2025-10-10T01:05:25.6578397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6578489Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6578775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6578912Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6579197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6579278Z self_outputs = self.self( 2025-10-10T01:05:25.6579564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6579740Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6580016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6580089Z x = self.pointwise(x) 2025-10-10T01:05:25.6580092Z 2025-10-10T01:05:25.6580204Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6580404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6580476Z res = mod(**inputs) 2025-10-10T01:05:25.6580756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6580838Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6581127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6581204Z hidden_states = self.encoder( 2025-10-10T01:05:25.6581496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6581571Z layer_outputs = layer_module( 2025-10-10T01:05:25.6581809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6581901Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6582188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6582286Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6582567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6582664Z self_outputs = self.self( 2025-10-10T01:05:25.6582957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6583121Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6583125Z 2025-10-10T01:05:25.6583246Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6583459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6583537Z res = mod(**inputs) 2025-10-10T01:05:25.6583841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6583932Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6584227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6584307Z hidden_states = self.encoder( 2025-10-10T01:05:25.6584599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6584675Z layer_outputs = layer_module( 2025-10-10T01:05:25.6584912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6585005Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6585290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6585424Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6585708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6585790Z self_outputs = self.self( 2025-10-10T01:05:25.6586076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6586281Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6586288Z 2025-10-10T01:05:25.6586417Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6586632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6586714Z res = mod(**inputs) 2025-10-10T01:05:25.6587012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6587110Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6587416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6587495Z hidden_states = self.encoder( 2025-10-10T01:05:25.6587793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6587870Z layer_outputs = layer_module( 2025-10-10T01:05:25.6588116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6588201Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6588490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6588582Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6588855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6588937Z self_outputs = self.self( 2025-10-10T01:05:25.6589207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6589363Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6589367Z 2025-10-10T01:05:25.6589456Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6589537Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6589648Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6589846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6589913Z res = mod(**inputs) 2025-10-10T01:05:25.6590187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6590288Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6590566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6590640Z hidden_states = self.encoder( 2025-10-10T01:05:25.6590920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6590992Z layer_outputs = layer_module( 2025-10-10T01:05:25.6591217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6591304Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6591574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6591664Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6591959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6592049Z self_outputs = self.self( 2025-10-10T01:05:25.6592327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6592445Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6592449Z 2025-10-10T01:05:25.6592562Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6592761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6592835Z res = mod(**inputs) 2025-10-10T01:05:25.6593105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6593188Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6593466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6593541Z hidden_states = self.encoder( 2025-10-10T01:05:25.6593818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6593894Z layer_outputs = layer_module( 2025-10-10T01:05:25.6594118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6594208Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6594476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6594567Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6594837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6594981Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6595257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6595340Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6595363Z 2025-10-10T01:05:25.6595476Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6595668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6595740Z res = mod(**inputs) 2025-10-10T01:05:25.6596004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6596083Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6596354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6596450Z hidden_states = self.encoder( 2025-10-10T01:05:25.6596727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6596799Z layer_outputs = layer_module( 2025-10-10T01:05:25.6597030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6597110Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6597379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6597473Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6597745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6597836Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6598175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6598332Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6598610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6598696Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6598700Z 2025-10-10T01:05:25.6598811Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6599022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6599095Z res = mod(**inputs) 2025-10-10T01:05:25.6599356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6599437Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6599708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6599781Z hidden_states = self.encoder( 2025-10-10T01:05:25.6600051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6600123Z layer_outputs = layer_module( 2025-10-10T01:05:25.6600342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6600427Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6600687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6600777Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6601029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6601116Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6601409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6601529Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6601824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6601937Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6602157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6602229Z return self.act(input) 2025-10-10T01:05:25.6602232Z 2025-10-10T01:05:25.6602336Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6602539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6602624Z res = mod(**inputs) 2025-10-10T01:05:25.6602893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6602975Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6603237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6603317Z hidden_states = self.encoder( 2025-10-10T01:05:25.6603578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6603655Z layer_outputs = layer_module( 2025-10-10T01:05:25.6603876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6603963Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6604252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6604356Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6604628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6604707Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6605018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6605154Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6605429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6605521Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6605525Z 2025-10-10T01:05:25.6605630Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6605845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6605912Z res = mod(**inputs) 2025-10-10T01:05:25.6606191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6606275Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6606549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6606630Z hidden_states = self.encoder( 2025-10-10T01:05:25.6606902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6606981Z layer_outputs = layer_module( 2025-10-10T01:05:25.6607207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6607292Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6607573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6607655Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6607948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6608021Z self_outputs = self.self( 2025-10-10T01:05:25.6608296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6608391Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6608395Z 2025-10-10T01:05:25.6608500Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6608717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6608785Z res = mod(**inputs) 2025-10-10T01:05:25.6609075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6609157Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6609424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6609506Z hidden_states = self.encoder( 2025-10-10T01:05:25.6609777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6609858Z layer_outputs = layer_module( 2025-10-10T01:05:25.6610082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6610169Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6610457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6610557Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6610837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6610911Z self_outputs = self.self( 2025-10-10T01:05:25.6611189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6611277Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6611280Z 2025-10-10T01:05:25.6611385Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6611591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6611658Z res = mod(**inputs) 2025-10-10T01:05:25.6611938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6612022Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6612299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6612377Z hidden_states = self.encoder( 2025-10-10T01:05:25.6612684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6612769Z layer_outputs = layer_module( 2025-10-10T01:05:25.6613016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6613107Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6613401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6613487Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6613793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6613870Z self_outputs = self.self( 2025-10-10T01:05:25.6614172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6614295Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6614299Z 2025-10-10T01:05:25.6614392Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6614478Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6614590Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6614807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6614877Z res = mod(**inputs) 2025-10-10T01:05:25.6615179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6615289Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6615585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6615669Z hidden_states = self.encoder( 2025-10-10T01:05:25.6616016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6616095Z layer_outputs = layer_module( 2025-10-10T01:05:25.6616319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6616400Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6616693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6616780Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6617121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6617199Z self_outputs = self.self( 2025-10-10T01:05:25.6617504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6617619Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6617623Z 2025-10-10T01:05:25.6617708Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6617829Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6618044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6618122Z res = mod(**inputs) 2025-10-10T01:05:25.6618465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6618548Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6618831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6618905Z hidden_states = self.encoder( 2025-10-10T01:05:25.6619197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6619276Z layer_outputs = layer_module( 2025-10-10T01:05:25.6619520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6619613Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6619909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6620002Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6620301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6620384Z self_outputs = self.self( 2025-10-10T01:05:25.6620679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6620872Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6621168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6621252Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6621257Z 2025-10-10T01:05:25.6621374Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6621585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6621658Z res = mod(**inputs) 2025-10-10T01:05:25.6621983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6622074Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6622369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6622448Z hidden_states = self.encoder( 2025-10-10T01:05:25.6622752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6622830Z layer_outputs = layer_module( 2025-10-10T01:05:25.6623068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6623160Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6623459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6623555Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6623889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6623967Z self_outputs = self.self( 2025-10-10T01:05:25.6624273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6624445Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6624737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6624813Z x = self.pointwise(x) 2025-10-10T01:05:25.6624817Z 2025-10-10T01:05:25.6624938Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6625147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6625217Z res = mod(**inputs) 2025-10-10T01:05:25.6625515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6625603Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6625893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6625972Z hidden_states = self.encoder( 2025-10-10T01:05:25.6626327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6626418Z layer_outputs = layer_module( 2025-10-10T01:05:25.6626655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6626749Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6627040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6627138Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6627421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6627525Z self_outputs = self.self( 2025-10-10T01:05:25.6627820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6627985Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6627989Z 2025-10-10T01:05:25.6628111Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6628323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6628395Z res = mod(**inputs) 2025-10-10T01:05:25.6628726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6628815Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6629109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6629187Z hidden_states = self.encoder( 2025-10-10T01:05:25.6629479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6629554Z layer_outputs = layer_module( 2025-10-10T01:05:25.6629790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6629882Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6630184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6630278Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6630607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6630685Z self_outputs = self.self( 2025-10-10T01:05:25.6631013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6631143Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6631147Z 2025-10-10T01:05:25.6631265Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6631612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6631708Z res = mod(**inputs) 2025-10-10T01:05:25.6632010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6632100Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6632402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6632481Z hidden_states = self.encoder( 2025-10-10T01:05:25.6632787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6632865Z layer_outputs = layer_module( 2025-10-10T01:05:25.6633100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6633194Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6633494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6633590Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6633876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6633955Z self_outputs = self.self( 2025-10-10T01:05:25.6634248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6634434Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6634439Z 2025-10-10T01:05:25.6634533Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6634618Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6634738Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6634950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6635021Z res = mod(**inputs) 2025-10-10T01:05:25.6635317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6635404Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6635731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6635811Z hidden_states = self.encoder( 2025-10-10T01:05:25.6636101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6636189Z layer_outputs = layer_module( 2025-10-10T01:05:25.6636426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6636518Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6636805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6636900Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6637195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6637289Z self_outputs = self.self( 2025-10-10T01:05:25.6637548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6637660Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6637663Z 2025-10-10T01:05:25.6637771Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6637961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6638023Z res = mod(**inputs) 2025-10-10T01:05:25.6638284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6638362Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6638628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6638698Z hidden_states = self.encoder( 2025-10-10T01:05:25.6638961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6639029Z layer_outputs = layer_module( 2025-10-10T01:05:25.6639244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6639329Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6639582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6639668Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6639924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6640050Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6640318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6640401Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6640405Z 2025-10-10T01:05:25.6640534Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6640723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6640796Z res = mod(**inputs) 2025-10-10T01:05:25.6641051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6641132Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6641397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6641467Z hidden_states = self.encoder( 2025-10-10T01:05:25.6641751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6641822Z layer_outputs = layer_module( 2025-10-10T01:05:25.6642032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6642118Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6642370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6642461Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6642710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6642785Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6643099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6643232Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6643495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6643577Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6643580Z 2025-10-10T01:05:25.6643687Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6643878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6643941Z res = mod(**inputs) 2025-10-10T01:05:25.6644207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6644286Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6644547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6644617Z hidden_states = self.encoder( 2025-10-10T01:05:25.6644870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6644948Z layer_outputs = layer_module( 2025-10-10T01:05:25.6645162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6645245Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6645504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6645597Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6645859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6645937Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6646251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6646381Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6646675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6646821Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6647053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6647143Z return self.act(input) 2025-10-10T01:05:25.6647147Z 2025-10-10T01:05:25.6647262Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6647487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6647561Z res = mod(**inputs) 2025-10-10T01:05:25.6647879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6647972Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6648258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6648356Z hidden_states = self.encoder( 2025-10-10T01:05:25.6648626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6648707Z layer_outputs = layer_module( 2025-10-10T01:05:25.6648930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6649010Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6649290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6650288Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6650577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6650662Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6650993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6651129Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6651398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6651499Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6651503Z 2025-10-10T01:05:25.6651605Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6651807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6651875Z res = mod(**inputs) 2025-10-10T01:05:25.6652141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6652238Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6652522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6652607Z hidden_states = self.encoder( 2025-10-10T01:05:25.6652891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6652973Z layer_outputs = layer_module( 2025-10-10T01:05:25.6653210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6653293Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6653587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6653673Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6653961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6654060Z self_outputs = self.self( 2025-10-10T01:05:25.6654344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-10-10T01:05:25.6654451Z mixed_query_layer = self.query(hidden_states) 2025-10-10T01:05:25.6654455Z 2025-10-10T01:05:25.6654567Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6654784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6654854Z res = mod(**inputs) 2025-10-10T01:05:25.6655164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6655256Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6655538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6655624Z hidden_states = self.encoder( 2025-10-10T01:05:25.6655907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6655990Z layer_outputs = layer_module( 2025-10-10T01:05:25.6656226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6656311Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6656601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6656729Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6657024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6657102Z self_outputs = self.self( 2025-10-10T01:05:25.6657396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-10-10T01:05:25.6657486Z mixed_key_layer = self.key(hidden_states) 2025-10-10T01:05:25.6657489Z 2025-10-10T01:05:25.6657600Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6657818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6657889Z res = mod(**inputs) 2025-10-10T01:05:25.6658184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6658277Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6658564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6658649Z hidden_states = self.encoder( 2025-10-10T01:05:25.6658936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6659020Z layer_outputs = layer_module( 2025-10-10T01:05:25.6659257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6659340Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6659621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6659701Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6659967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6660036Z self_outputs = self.self( 2025-10-10T01:05:25.6660307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-10-10T01:05:25.6660419Z mixed_value_layer = self.value(hidden_states) 2025-10-10T01:05:25.6660422Z 2025-10-10T01:05:25.6660503Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6660591Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6660694Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6660894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6660958Z res = mod(**inputs) 2025-10-10T01:05:25.6661219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6661325Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6661594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6661675Z hidden_states = self.encoder( 2025-10-10T01:05:25.6661943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6662023Z layer_outputs = layer_module( 2025-10-10T01:05:25.6662247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6662326Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6662605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6662687Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6662983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6663077Z self_outputs = self.self( 2025-10-10T01:05:25.6663357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-10-10T01:05:25.6663479Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-10-10T01:05:25.6663483Z 2025-10-10T01:05:25.6663569Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6663689Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6663901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6663970Z res = mod(**inputs) 2025-10-10T01:05:25.6664265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6664353Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6664649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6664726Z hidden_states = self.encoder( 2025-10-10T01:05:25.6665020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6665099Z layer_outputs = layer_module( 2025-10-10T01:05:25.6665336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6665428Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6665728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6665824Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6666122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6666270Z self_outputs = self.self( 2025-10-10T01:05:25.6666576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6666749Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6667080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-10-10T01:05:25.6667168Z x = self.depthwise(hidden_states) 2025-10-10T01:05:25.6667172Z 2025-10-10T01:05:25.6667297Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6667513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6667587Z res = mod(**inputs) 2025-10-10T01:05:25.6667889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6668001Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6668304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6668384Z hidden_states = self.encoder( 2025-10-10T01:05:25.6668671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6668749Z layer_outputs = layer_module( 2025-10-10T01:05:25.6668960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6669046Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6669301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6669387Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6669678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6669750Z self_outputs = self.self( 2025-10-10T01:05:25.6670010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-10-10T01:05:25.6670163Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-10-10T01:05:25.6670428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-10-10T01:05:25.6670496Z x = self.pointwise(x) 2025-10-10T01:05:25.6670499Z 2025-10-10T01:05:25.6670600Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6670815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6670879Z res = mod(**inputs) 2025-10-10T01:05:25.6671145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6671224Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6671487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6671557Z hidden_states = self.encoder( 2025-10-10T01:05:25.6671812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6671890Z layer_outputs = layer_module( 2025-10-10T01:05:25.6672101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6672183Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6672437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6672520Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6672782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6672852Z self_outputs = self.self( 2025-10-10T01:05:25.6673132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-10-10T01:05:25.6673280Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-10-10T01:05:25.6673283Z 2025-10-10T01:05:25.6673392Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6673583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6673647Z res = mod(**inputs) 2025-10-10T01:05:25.6673910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6674021Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6674284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6674354Z hidden_states = self.encoder( 2025-10-10T01:05:25.6674621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6674698Z layer_outputs = layer_module( 2025-10-10T01:05:25.6674917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6675001Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6675270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6675348Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6675634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6675723Z self_outputs = self.self( 2025-10-10T01:05:25.6675984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-10-10T01:05:25.6676102Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-10-10T01:05:25.6676106Z 2025-10-10T01:05:25.6676214Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6676402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6676466Z res = mod(**inputs) 2025-10-10T01:05:25.6676729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6676807Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6677076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6677150Z hidden_states = self.encoder( 2025-10-10T01:05:25.6677411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6677489Z layer_outputs = layer_module( 2025-10-10T01:05:25.6677715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6677799Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6678054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6678138Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6678400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6678475Z self_outputs = self.self( 2025-10-10T01:05:25.6678745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-10-10T01:05:25.6678871Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-10-10T01:05:25.6678894Z 2025-10-10T01:05:25.6678981Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6679059Z cudagraph partition due to non gpu ops 2025-10-10T01:05:25.6679162Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6679365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6679433Z res = mod(**inputs) 2025-10-10T01:05:25.6679706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6679787Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6680078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6680153Z hidden_states = self.encoder( 2025-10-10T01:05:25.6680420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6680498Z layer_outputs = layer_module( 2025-10-10T01:05:25.6680717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6680802Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6681067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6681146Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6681425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-10-10T01:05:25.6681536Z self_outputs = self.self( 2025-10-10T01:05:25.6681815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-10-10T01:05:25.6681932Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-10-10T01:05:25.6681937Z 2025-10-10T01:05:25.6682050Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6682248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6682313Z res = mod(**inputs) 2025-10-10T01:05:25.6682588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6682669Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6682944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6683021Z hidden_states = self.encoder( 2025-10-10T01:05:25.6683296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6683372Z layer_outputs = layer_module( 2025-10-10T01:05:25.6683590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6683671Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6683933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-10-10T01:05:25.6684013Z self_attention_outputs = self.attention( 2025-10-10T01:05:25.6684283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-10-10T01:05:25.6684413Z attention_output = self.output(self_outputs[0], hidden_states) 2025-10-10T01:05:25.6684684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-10-10T01:05:25.6684768Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6684772Z 2025-10-10T01:05:25.6684880Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6685094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6685160Z res = mod(**inputs) 2025-10-10T01:05:25.6685431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6685511Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6685778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6685849Z hidden_states = self.encoder( 2025-10-10T01:05:25.6686135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6686215Z layer_outputs = layer_module( 2025-10-10T01:05:25.6686432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6686519Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6686779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6686871Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6687127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6687206Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6687507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6687664Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6687941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-10-10T01:05:25.6688025Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6688030Z 2025-10-10T01:05:25.6688133Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6688337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6688404Z res = mod(**inputs) 2025-10-10T01:05:25.6688677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6688759Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6689031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6689108Z hidden_states = self.encoder( 2025-10-10T01:05:25.6689372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6689452Z layer_outputs = layer_module( 2025-10-10T01:05:25.6689672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6689758Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6690020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6690103Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6690368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6690446Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6690751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-10-10T01:05:25.6690873Z intermediate_output = self.intermediate(attention_output) 2025-10-10T01:05:25.6691144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-10-10T01:05:25.6691278Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-10-10T01:05:25.6691491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-10-10T01:05:25.6691570Z return self.act(input) 2025-10-10T01:05:25.6691574Z 2025-10-10T01:05:25.6691677Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6691880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6691945Z res = mod(**inputs) 2025-10-10T01:05:25.6692226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-10-10T01:05:25.6692321Z generator_hidden_states = self.convbert( 2025-10-10T01:05:25.6692580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-10-10T01:05:25.6692659Z hidden_states = self.encoder( 2025-10-10T01:05:25.6692921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-10-10T01:05:25.6692997Z layer_outputs = layer_module( 2025-10-10T01:05:25.6693216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-10-10T01:05:25.6693294Z return super().__call__(*args, **kwargs) 2025-10-10T01:05:25.6693562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-10-10T01:05:25.6693682Z layer_output = apply_chunking_to_forward( 2025-10-10T01:05:25.6693964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-10-10T01:05:25.6694041Z return forward_fn(*input_tensors) 2025-10-10T01:05:25.6694336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-10-10T01:05:25.6694485Z layer_output = self.output(intermediate_output, attention_output) 2025-10-10T01:05:25.6694741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-10-10T01:05:25.6694829Z hidden_states = self.dense(hidden_states) 2025-10-10T01:05:25.6694832Z 2025-10-10T01:05:25.6694933Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6695131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6695201Z res = mod(**inputs) 2025-10-10T01:05:25.6695465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-10-10T01:05:25.6695624Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-10-10T01:05:25.6695891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-10-10T01:05:25.6696002Z hidden_states = self.dense(generator_hidden_states) 2025-10-10T01:05:25.6696006Z 2025-10-10T01:05:25.6696108Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6696300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6696375Z res = mod(**inputs) 2025-10-10T01:05:25.6696640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-10-10T01:05:25.6696778Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-10-10T01:05:25.6696782Z 2025-10-10T01:05:25.6696885Z cudagraph partition due to non gpu ops. Found from : 2025-10-10T01:05:25.6697084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-10-10T01:05:25.6697180Z res = mod(**inputs) 2025-10-10T01:05:25.6697448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-10-10T01:05:25.6697622Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-10-10T01:05:25.6697626Z 2025-10-10T01:05:37.0599141Z Compilation time (from dynamo_timed): 23.123297451 2025-10-10T01:05:37.0644676Z pass 2025-10-10T01:05:37.0645119Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-10-10T01:05:37.0646487Z TIMING: _recursive_pre_grad_passes:0.01077 _recursive_joint_graph_passes:0.65902 _recursive_post_grad_passes:0.18104 async_compile.wait:0.69973 code_gen:10.86176 inductor_compile:13.5044 backend_compile:18.58508 gc:0.0023 entire_frame_compile:23.1233 total_wall_time:23.1233 2025-10-10T01:05:37.0647654Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:13782 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:3940 2025-10-10T01:05:37.0648325Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-10-10T01:05:39.4383232Z accuracy pass_rate=92.59% 2025-10-10T01:05:39.4384479Z calls_captured gmean=0.00x mean=579.519x 2025-10-10T01:05:39.4385782Z unique_graphs gmean=0.00x mean=1.111x 2025-10-10T01:05:39.4390775Z graph_breaks gmean=0.00x mean=0.222x 2025-10-10T01:05:39.4402378Z unique_graph_breaks gmean=0.00x mean=0.074x 2025-10-10T01:05:39.4406627Z autograd_captures gmean=0.00x mean=0.000x 2025-10-10T01:05:39.4411745Z autograd_compiles gmean=0.00x mean=0.000x 2025-10-10T01:05:39.4416958Z cudagraph_skips gmean=0.00x mean=1.111x 2025-10-10T01:05:39.4421712Z compilation_latency mean=21.888 seconds 2025-10-10T01:05:40.4828760Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-10-10T01:05:40.7867669Z AlbertForMaskedLM PASS 2025-10-10T01:05:40.7867983Z AllenaiLongformerBase PASS 2025-10-10T01:05:40.7876404Z BartForCausalLM PASS 2025-10-10T01:05:40.7880659Z BertForMaskedLM PASS 2025-10-10T01:05:40.7886508Z BlenderbotForCausalLM XFAIL 2025-10-10T01:05:40.7886827Z DebertaV2ForMaskedLM XFAIL 2025-10-10T01:05:40.7888950Z DistilBertForMaskedLM PASS 2025-10-10T01:05:40.7889205Z DistillGPT2 PASS 2025-10-10T01:05:40.7893017Z ElectraForCausalLM PASS 2025-10-10T01:05:40.7893388Z GPT2ForSequenceClassification PASS 2025-10-10T01:05:40.7899920Z GoogleFnet PASS 2025-10-10T01:05:40.7900171Z LayoutLMForMaskedLM PASS 2025-10-10T01:05:40.7904917Z M2M100ForConditionalGeneration PASS 2025-10-10T01:05:40.7910521Z MBartForCausalLM PASS 2025-10-10T01:05:40.7917343Z MT5ForConditionalGeneration PASS 2025-10-10T01:05:40.7917804Z MegatronBertForCausalLM PASS 2025-10-10T01:05:40.7923381Z MobileBertForMaskedLM PASS 2025-10-10T01:05:40.7923691Z OPTForCausalLM PASS 2025-10-10T01:05:40.7930300Z PLBartForCausalLM PASS 2025-10-10T01:05:40.7930823Z PegasusForCausalLM PASS 2025-10-10T01:05:40.7937076Z RobertaForCausalLM PASS 2025-10-10T01:05:40.7937784Z T5ForConditionalGeneration PASS 2025-10-10T01:05:40.7942864Z T5Small PASS 2025-10-10T01:05:40.7944129Z TrOCRForCausalLM PASS 2025-10-10T01:05:40.7946754Z XGLMForCausalLM PASS 2025-10-10T01:05:40.7955369Z XLNetLMHeadModel PASS 2025-10-10T01:05:40.7961684Z YituTechConvBert PASS 2025-10-10T01:05:40.8499368Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-10-10T01:05:41.1247497Z AlbertForMaskedLM PASS 2025-10-10T01:05:41.1248978Z AllenaiLongformerBase PASS 2025-10-10T01:05:41.1249295Z BartForCausalLM PASS 2025-10-10T01:05:41.1249627Z BertForMaskedLM PASS 2025-10-10T01:05:41.1249884Z BlenderbotForCausalLM PASS 2025-10-10T01:05:41.1260780Z DebertaV2ForMaskedLM PASS 2025-10-10T01:05:41.1261106Z DistilBertForMaskedLM PASS 2025-10-10T01:05:41.1261330Z DistillGPT2 PASS 2025-10-10T01:05:41.1264071Z ElectraForCausalLM PASS 2025-10-10T01:05:41.1266202Z GPT2ForSequenceClassification PASS 2025-10-10T01:05:41.1266663Z GoogleFnet PASS 2025-10-10T01:05:41.1266936Z LayoutLMForMaskedLM PASS 2025-10-10T01:05:41.1278422Z M2M100ForConditionalGeneration PASS 2025-10-10T01:05:41.1283224Z MBartForCausalLM PASS 2025-10-10T01:05:41.1288016Z MT5ForConditionalGeneration PASS 2025-10-10T01:05:41.1294419Z MegatronBertForCausalLM PASS 2025-10-10T01:05:41.1299270Z MobileBertForMaskedLM PASS 2025-10-10T01:05:41.1301431Z OPTForCausalLM PASS 2025-10-10T01:05:41.1301674Z PLBartForCausalLM PASS 2025-10-10T01:05:41.1301912Z PegasusForCausalLM PASS 2025-10-10T01:05:41.1302147Z RobertaForCausalLM PASS 2025-10-10T01:05:41.1302379Z T5ForConditionalGeneration PASS 2025-10-10T01:05:41.1306862Z T5Small PASS 2025-10-10T01:05:41.1307169Z TrOCRForCausalLM PASS 2025-10-10T01:05:41.1313071Z XGLMForCausalLM PASS_BUT_FLAKY 2025-10-10T01:05:41.1318363Z XLNetLMHeadModel PASS 2025-10-10T01:05:41.1318657Z YituTechConvBert PASS 2025-10-10T01:05:41.1818048Z + sccache_epilogue 2025-10-10T01:05:41.1818394Z + echo '::group::Sccache Compilation Log' 2025-10-10T01:05:41.1818950Z ##[group]Sccache Compilation Log 2025-10-10T01:05:41.1819228Z + echo '=================== sccache compilation log ===================' 2025-10-10T01:05:41.1819532Z =================== sccache compilation log =================== 2025-10-10T01:05:41.1820009Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-10-10T01:05:41.2045417Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-10-10T01:05:41.2045989Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-10-10T01:05:41.2046403Z + sccache --show-stats 2025-10-10T01:05:41.2077311Z Compile requests 276 2025-10-10T01:05:41.2077642Z Compile requests executed 0 2025-10-10T01:05:41.2077902Z Cache hits 0 2025-10-10T01:05:41.2078122Z Cache misses 0 2025-10-10T01:05:41.2078348Z Cache hits rate - 2025-10-10T01:05:41.2078568Z Cache timeouts 0 2025-10-10T01:05:41.2078776Z Cache read errors 0 2025-10-10T01:05:41.2079001Z Forced recaches 0 2025-10-10T01:05:41.2079227Z Cache write errors 0 2025-10-10T01:05:41.2079463Z Cache errors 0 2025-10-10T01:05:41.2079717Z Compilations 0 2025-10-10T01:05:41.2079956Z Compilation failures 0 2025-10-10T01:05:41.2080180Z Non-cacheable compilations 0 2025-10-10T01:05:41.2080423Z Non-cacheable calls 25 2025-10-10T01:05:41.2080643Z Non-compilation calls 251 2025-10-10T01:05:41.2080861Z Unsupported compiler calls 0 2025-10-10T01:05:41.2081090Z Average cache write 0.000 s 2025-10-10T01:05:41.2081557Z Average compiler 0.000 s 2025-10-10T01:05:41.2081790Z Average cache read hit 0.000 s 2025-10-10T01:05:41.2082021Z Failed distributed compilations 0 2025-10-10T01:05:41.2082179Z 2025-10-10T01:05:41.2082259Z Non-cacheable reasons: 2025-10-10T01:05:41.2082465Z -E 25 2025-10-10T01:05:41.2082610Z 2025-10-10T01:05:41.2082796Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-10-10T01:05:41.2083118Z Version (client) 0.10.0 2025-10-10T01:05:41.2083347Z + sccache --stop-server 2025-10-10T01:05:41.2097118Z Stopping sccache server... 2025-10-10T01:05:41.2097412Z Compile requests 276 2025-10-10T01:05:41.2097919Z Compile requests executed 0 2025-10-10T01:05:41.2098193Z Cache hits 0 2025-10-10T01:05:41.2098418Z Cache misses 0 2025-10-10T01:05:41.2098654Z Cache hits rate - 2025-10-10T01:05:41.2098887Z Cache timeouts 0 2025-10-10T01:05:41.2099115Z Cache read errors 0 2025-10-10T01:05:41.2099340Z Forced recaches 0 2025-10-10T01:05:41.2099566Z Cache write errors 0 2025-10-10T01:05:41.2099786Z Cache errors 0 2025-10-10T01:05:41.2100006Z Compilations 0 2025-10-10T01:05:41.2100233Z Compilation failures 0 2025-10-10T01:05:41.2100469Z Non-cacheable compilations 0 2025-10-10T01:05:41.2100700Z Non-cacheable calls 25 2025-10-10T01:05:41.2100927Z Non-compilation calls 251 2025-10-10T01:05:41.2101162Z Unsupported compiler calls 0 2025-10-10T01:05:41.2101517Z Average cache write 0.000 s 2025-10-10T01:05:41.2101763Z Average compiler 0.000 s 2025-10-10T01:05:41.2101995Z Average cache read hit 0.000 s 2025-10-10T01:05:41.2102242Z Failed distributed compilations 0 2025-10-10T01:05:41.2102403Z 2025-10-10T01:05:41.2102487Z Non-cacheable reasons: 2025-10-10T01:05:41.2102689Z -E 25 2025-10-10T01:05:41.2102829Z 2025-10-10T01:05:41.2103012Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-10-10T01:05:41.2103331Z Version (client) 0.10.0 2025-10-10T01:05:41.2103615Z + echo ::endgroup:: 2025-10-10T01:05:41.2104235Z ##[endgroup] 2025-10-10T01:05:41.2104418Z + cleanup_workspace 2025-10-10T01:05:41.2104769Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-10-10T01:05:41.2105280Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-10-10T01:05:41.2105708Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-10-10T01:05:41.2106034Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-10-10T01:05:41.2106782Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-10-10T01:05:41.2107202Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-10-10T01:05:41.2107541Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-10-10T01:05:41.6650952Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-10-10T01:05:41.6651261Z with: 2025-10-10T01:05:41.6651456Z benchmark-results-dir: test/test-reports 2025-10-10T01:05:41.6651690Z dry-run: false 2025-10-10T01:05:41.6651870Z schema-version: v3 2025-10-10T01:05:41.6652250Z github-token: *** 2025-10-10T01:05:41.6652418Z env: 2025-10-10T01:05:41.6652585Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:41.6653040Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:41.6653381Z ##[endgroup] 2025-10-10T01:05:41.6681550Z ##[group]Run set -eux 2025-10-10T01:05:41.6681772Z set -eux 2025-10-10T01:05:41.6681923Z  2025-10-10T01:05:41.6682082Z if [[ -n "" ]]; then 2025-10-10T01:05:41.6682343Z  source "" 2025-10-10T01:05:41.6682509Z fi 2025-10-10T01:05:41.6682742Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-10-10T01:05:41.6683053Z  2025-10-10T01:05:41.6683212Z DEVICE_NAME="" 2025-10-10T01:05:41.6683390Z DEVICE_TYPE="" 2025-10-10T01:05:41.6683559Z  2025-10-10T01:05:41.6683722Z if command -v nvidia-smi; then 2025-10-10T01:05:41.6684005Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-10-10T01:05:41.6684341Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-10-10T01:05:41.6684671Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-10-10T01:05:41.6684949Z  python3 -mpip install torch==2.7.1 2025-10-10T01:05:41.6685188Z elif command -v rocminfo; then 2025-10-10T01:05:41.6685473Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-10-10T01:05:41.6685829Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-10-10T01:05:41.6686179Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-10-10T01:05:41.6686496Z  # GPU device name coming from rocminfo instead 2025-10-10T01:05:41.6686737Z  DEVICE_NAME=rocm 2025-10-10T01:05:41.6687052Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-10-10T01:05:41.6687365Z fi 2025-10-10T01:05:41.6687527Z  2025-10-10T01:05:41.6687723Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-10-10T01:05:41.6688042Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-10-10T01:05:41.6696461Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:41.6696713Z env: 2025-10-10T01:05:41.6696881Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:41.6697191Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:41.6697489Z ##[endgroup] 2025-10-10T01:05:41.6725225Z + [[ -n '' ]] 2025-10-10T01:05:41.6725588Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-10-10T01:05:41.8547370Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T01:05:42.6640924Z Collecting boto3==1.35.33 2025-10-10T01:05:42.6801814Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-10-10T01:05:42.9103224Z Collecting psutil==7.0.0 2025-10-10T01:05:42.9143390Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-10-10T01:05:42.9436889Z Collecting pynvml==12.0.0 2025-10-10T01:05:42.9473651Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-10-10T01:05:43.7864590Z Collecting botocore<1.36.0,>=1.35.33 2025-10-10T01:05:43.7914174Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-10-10T01:05:43.8881111Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-10-10T01:05:43.9226147Z Collecting s3transfer<0.11.0,>=0.10.0 2025-10-10T01:05:43.9274740Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-10-10T01:05:43.9891682Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-10-10T01:05:43.9929164Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-10-10T01:05:44.0010611Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-10-10T01:05:44.0018175Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-10-10T01:05:44.1654642Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-10-10T01:05:44.2786389Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-10-10T01:05:44.6480704Z Attempting uninstall: nvidia-ml-py 2025-10-10T01:05:44.6481139Z Found existing installation: nvidia-ml-py 11.525.84 2025-10-10T01:05:44.6490450Z Uninstalling nvidia-ml-py-11.525.84: 2025-10-10T01:05:44.6624012Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-10-10T01:05:44.7121714Z Attempting uninstall: psutil 2025-10-10T01:05:44.7122053Z Found existing installation: psutil 5.9.8 2025-10-10T01:05:44.7169809Z Uninstalling psutil-5.9.8: 2025-10-10T01:05:44.7174675Z Successfully uninstalled psutil-5.9.8 2025-10-10T01:05:44.8481192Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-10-10T01:05:44.9540146Z + DEVICE_NAME= 2025-10-10T01:05:44.9540455Z + DEVICE_TYPE= 2025-10-10T01:05:44.9540939Z + command -v nvidia-smi 2025-10-10T01:05:44.9541232Z + command -v rocminfo 2025-10-10T01:05:44.9544733Z + echo DEVICE_NAME= 2025-10-10T01:05:44.9544949Z + echo DEVICE_TYPE= 2025-10-10T01:05:44.9573747Z ##[group]Run set -eux 2025-10-10T01:05:44.9573950Z set -eux 2025-10-10T01:05:44.9574175Z  2025-10-10T01:05:44.9574347Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-10-10T01:05:44.9574582Z  echo "Missing github-token input" 2025-10-10T01:05:44.9574791Z  exit 1 2025-10-10T01:05:44.9574949Z fi 2025-10-10T01:05:44.9579577Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:44.9579821Z env: 2025-10-10T01:05:44.9579982Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:44.9580287Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:44.9580681Z DEVICE_NAME: 2025-10-10T01:05:44.9580838Z DEVICE_TYPE: 2025-10-10T01:05:44.9581218Z GITHUB_TOKEN: *** 2025-10-10T01:05:44.9581384Z ##[endgroup] 2025-10-10T01:05:44.9604977Z + [[ -z *** ]] 2025-10-10T01:05:44.9646901Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-10-10T01:05:44.9647203Z with: 2025-10-10T01:05:44.9647508Z github-token: *** 2025-10-10T01:05:44.9647683Z env: 2025-10-10T01:05:44.9647865Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:44.9648178Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:44.9648512Z DEVICE_NAME: 2025-10-10T01:05:44.9648688Z DEVICE_TYPE: 2025-10-10T01:05:44.9648861Z ##[endgroup] 2025-10-10T01:05:44.9667147Z ##[group]Run set -eux 2025-10-10T01:05:44.9667367Z set -eux 2025-10-10T01:05:44.9667541Z  2025-10-10T01:05:44.9667860Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-10-10T01:05:44.9672323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:44.9672568Z env: 2025-10-10T01:05:44.9672728Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:44.9673031Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:44.9673343Z DEVICE_NAME: 2025-10-10T01:05:44.9673508Z DEVICE_TYPE: 2025-10-10T01:05:44.9673801Z GITHUB_TOKEN: *** 2025-10-10T01:05:44.9674098Z ##[endgroup] 2025-10-10T01:05:44.9697958Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 18392306145 i-01dad1d3a0e80f7c0 2025-10-10T01:05:46.0128510Z setting job-id=52406535390 2025-10-10T01:05:46.0131732Z setting job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T01:05:46.0243555Z ##[group]Run set -eux 2025-10-10T01:05:46.0243786Z set -eux 2025-10-10T01:05:46.0243955Z  2025-10-10T01:05:46.0244123Z if [[ -n "" ]]; then 2025-10-10T01:05:46.0244316Z  source "" 2025-10-10T01:05:46.0244477Z fi 2025-10-10T01:05:46.0244630Z  2025-10-10T01:05:46.0244893Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-10-10T01:05:46.0245311Z  --schema-version "${SCHEMA_VERSION}" \ 2025-10-10T01:05:46.0245539Z  --repo "${REPO}" \ 2025-10-10T01:05:46.0245763Z  --head-branch "${HEAD_BRANCH}" \ 2025-10-10T01:05:46.0245989Z  --head-sha "${HEAD_SHA}" \ 2025-10-10T01:05:46.0246221Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-10-10T01:05:46.0246454Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-10-10T01:05:46.0246672Z  --job-id "${JOB_ID}" \ 2025-10-10T01:05:46.0246883Z  --job-name "${JOB_NAME}" 2025-10-10T01:05:46.0251667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.0251920Z env: 2025-10-10T01:05:46.0252073Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.0252380Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.0252698Z DEVICE_NAME: 2025-10-10T01:05:46.0252865Z DEVICE_TYPE: 2025-10-10T01:05:46.0253029Z SCHEMA_VERSION: v3 2025-10-10T01:05:46.0253208Z REPO: pytorch/pytorch 2025-10-10T01:05:46.0253397Z HEAD_BRANCH: refs/heads/main 2025-10-10T01:05:46.0253628Z HEAD_SHA: 344e6365a0068c2d2847fcec0c55dd53291d475e 2025-10-10T01:05:46.0253855Z WORKFLOW_RUN_ID: 18392306145 2025-10-10T01:05:46.0254046Z RUN_ATTEMPT: 1 2025-10-10T01:05:46.0254200Z JOB_ID: 52406535390 2025-10-10T01:05:46.0254481Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T01:05:46.0254770Z ##[endgroup] 2025-10-10T01:05:46.0278518Z + [[ -n '' ]] 2025-10-10T01:05:46.0279930Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 344e6365a0068c2d2847fcec0c55dd53291d475e --workflow-id 18392306145 --run-attempt 1 --job-id 52406535390 --job-name 'inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-10-10T01:05:46.0543231Z ##[group]Run set -eux 2025-10-10T01:05:46.0543444Z set -eux 2025-10-10T01:05:46.0543613Z  2025-10-10T01:05:46.0543785Z if [[ -n "" ]]; then 2025-10-10T01:05:46.0543979Z  source "" 2025-10-10T01:05:46.0544148Z fi 2025-10-10T01:05:46.0544300Z  2025-10-10T01:05:46.0544559Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-10-10T01:05:46.0549606Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.0549851Z env: 2025-10-10T01:05:46.0550011Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.0550317Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.0550634Z DEVICE_NAME: 2025-10-10T01:05:46.0550798Z DEVICE_TYPE: 2025-10-10T01:05:46.0550959Z ##[endgroup] 2025-10-10T01:05:46.0574702Z + [[ -n '' ]] 2025-10-10T01:05:46.0577824Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-10-10T01:05:46.0902019Z INFO:root:Fail to import torch to get the device name 2025-10-10T01:05:46.0998470Z ##[group]Run set -eux 2025-10-10T01:05:46.0998704Z set -eux 2025-10-10T01:05:46.0998888Z  2025-10-10T01:05:46.0999085Z # TODO (huydhn): Implement this part 2025-10-10T01:05:46.0999373Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-10-10T01:05:46.1004466Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.1004734Z env: 2025-10-10T01:05:46.1004905Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.1005252Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.1005599Z DEVICE_NAME: 2025-10-10T01:05:46.1005782Z DEVICE_TYPE: 2025-10-10T01:05:46.1005960Z ##[endgroup] 2025-10-10T01:05:46.1026793Z + echo 'dependencies={}' 2025-10-10T01:05:46.1065132Z ##[group]Run set -eux 2025-10-10T01:05:46.1065356Z set -eux 2025-10-10T01:05:46.1065533Z  2025-10-10T01:05:46.1065707Z if [[ -n "" ]]; then 2025-10-10T01:05:46.1065915Z  source "" 2025-10-10T01:05:46.1066093Z fi 2025-10-10T01:05:46.1066423Z  2025-10-10T01:05:46.1066649Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-10-10T01:05:46.1066966Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-10-10T01:05:46.1067315Z  # We don't want the job to fail if the directory doesn't exist 2025-10-10T01:05:46.1067590Z  exit 0 2025-10-10T01:05:46.1067779Z fi 2025-10-10T01:05:46.1067940Z  2025-10-10T01:05:46.1068115Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-10-10T01:05:46.1068441Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-10-10T01:05:46.1068786Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-10-10T01:05:46.1069060Z  --metadata "${BENCHMARK_METADATA}" \ 2025-10-10T01:05:46.1069279Z  --runners "${RUNNER_INFO}" \ 2025-10-10T01:05:46.1069509Z  --dependencies "${DEPENDENCIES}" \ 2025-10-10T01:05:46.1069724Z  --dry-run 2025-10-10T01:05:46.1069895Z else 2025-10-10T01:05:46.1070131Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-10-10T01:05:46.1070463Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-10-10T01:05:46.1070724Z  --metadata "${BENCHMARK_METADATA}" \ 2025-10-10T01:05:46.1070946Z  --runners "${RUNNER_INFO}" \ 2025-10-10T01:05:46.1071228Z  --dependencies "${DEPENDENCIES}" 2025-10-10T01:05:46.1071433Z fi 2025-10-10T01:05:46.1075398Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.1075650Z env: 2025-10-10T01:05:46.1075809Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.1076119Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.1076432Z DEVICE_NAME: 2025-10-10T01:05:46.1076600Z DEVICE_TYPE: 2025-10-10T01:05:46.1076791Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-10-10T01:05:46.1076997Z DRY_RUN: false 2025-10-10T01:05:46.1077785Z BENCHMARK_METADATA: {"timestamp": 1760058346, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "344e6365a0068c2d2847fcec0c55dd53291d475e", "workflow_id": 18392306145, "run_attempt": 1, "job_id": 52406535390} 2025-10-10T01:05:46.1078781Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-32-50.ec2.internal"}, "name": "", "type": ""}] 2025-10-10T01:05:46.1079164Z DEPENDENCIES: {} 2025-10-10T01:05:46.1079337Z ##[endgroup] 2025-10-10T01:05:46.1099938Z + [[ -n '' ]] 2025-10-10T01:05:46.1100179Z + [[ ! -d test/test-reports ]] 2025-10-10T01:05:46.1100443Z + [[ false == \t\r\u\e ]] 2025-10-10T01:05:46.1102489Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1760058346, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "344e6365a0068c2d2847fcec0c55dd53291d475e", "workflow_id": 18392306145, "run_attempt": 1, "job_id": 52406535390}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-32-50.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-10-10T01:05:46.2220291Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/18392306145/52406535390/inference_huggingface.json 2025-10-10T01:05:46.2508276Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-10-10T01:05:46.4610857Z ##[group]Run cat test/**/*_toprint.log || true 2025-10-10T01:05:46.4611165Z cat test/**/*_toprint.log || true 2025-10-10T01:05:46.4616405Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.4616688Z env: 2025-10-10T01:05:46.4616866Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.4617201Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.4617537Z DEVICE_NAME: 2025-10-10T01:05:46.4617719Z DEVICE_TYPE: 2025-10-10T01:05:46.4617910Z ##[endgroup] 2025-10-10T01:05:46.4694748Z cat: 'test/**/*_toprint.log': No such file or directory 2025-10-10T01:05:46.4726768Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-10-10T01:05:46.4727048Z kill "$MONITOR_SCRIPT_PID" 2025-10-10T01:05:46.4731351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:46.4731794Z env: 2025-10-10T01:05:46.4731971Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:46.4732299Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:46.4732627Z DEVICE_NAME: 2025-10-10T01:05:46.4732805Z DEVICE_TYPE: 2025-10-10T01:05:46.4732991Z MONITOR_SCRIPT_PID: 48409 2025-10-10T01:05:46.4733192Z ##[endgroup] 2025-10-10T01:05:46.4835503Z Prepare all required actions 2025-10-10T01:05:46.4835850Z Getting action download info 2025-10-10T01:05:46.6265886Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-10-10T01:05:46.8547230Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-10-10T01:05:47.2222054Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-10-10T01:05:47.2222335Z with: 2025-10-10T01:05:47.2222651Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T01:05:47.2223022Z s3-bucket: gha-artifacts 2025-10-10T01:05:47.2223233Z env: 2025-10-10T01:05:47.2223415Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2223820Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2224174Z DEVICE_NAME: 2025-10-10T01:05:47.2224363Z DEVICE_TYPE: 2025-10-10T01:05:47.2224543Z ##[endgroup] 2025-10-10T01:05:47.2253814Z ##[group]Run # Remove any previous test jsons if they exist 2025-10-10T01:05:47.2254174Z # Remove any previous test jsons if they exist 2025-10-10T01:05:47.2254441Z rm -f test-jsons-*.zip 2025-10-10T01:05:47.2254758Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-10-10T01:05:47.2259977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:47.2260272Z env: 2025-10-10T01:05:47.2260525Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2260948Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2261404Z DEVICE_NAME: 2025-10-10T01:05:47.2261815Z DEVICE_TYPE: 2025-10-10T01:05:47.2262204Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T01:05:47.2262595Z ##[endgroup] 2025-10-10T01:05:47.2433020Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-10-10T01:05:47.2481714Z ##[group]Run # Remove any previous test reports if they exist 2025-10-10T01:05:47.2482049Z # Remove any previous test reports if they exist 2025-10-10T01:05:47.2482310Z rm -f test-reports-*.zip 2025-10-10T01:05:47.2482626Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-10-10T01:05:47.2487459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:47.2487711Z env: 2025-10-10T01:05:47.2487873Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2488175Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2488557Z DEVICE_NAME: 2025-10-10T01:05:47.2488715Z DEVICE_TYPE: 2025-10-10T01:05:47.2488987Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T01:05:47.2489281Z ##[endgroup] 2025-10-10T01:05:47.2536534Z adding: test/test-reports/inference_huggingface.csv (deflated 62%) 2025-10-10T01:05:47.2537524Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-10-10T01:05:47.2538004Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 64%) 2025-10-10T01:05:47.2563979Z ##[group]Run # Remove any previous usage logs if they exist 2025-10-10T01:05:47.2564334Z # Remove any previous usage logs if they exist 2025-10-10T01:05:47.2564600Z rm -f logs-*.zip 2025-10-10T01:05:47.2564861Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-10-10T01:05:47.2565199Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-10-10T01:05:47.2569862Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:47.2570130Z env: 2025-10-10T01:05:47.2570308Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2570634Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2570976Z DEVICE_NAME: 2025-10-10T01:05:47.2571157Z DEVICE_TYPE: 2025-10-10T01:05:47.2571602Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T01:05:47.2571930Z ##[endgroup] 2025-10-10T01:05:47.2630866Z adding: usage_log.txt (deflated 96%) 2025-10-10T01:05:47.2646758Z 2025-10-10T01:05:47.2650463Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip) 2025-10-10T01:05:47.2677489Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-10-10T01:05:47.2677839Z # Remove any previous debugging artifacts if they exist 2025-10-10T01:05:47.2678114Z rm -f debug-*.zip 2025-10-10T01:05:47.2678317Z if [ -d 'test/debug' ]; then 2025-10-10T01:05:47.2678554Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-10-10T01:05:47.2678780Z fi 2025-10-10T01:05:47.2683070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:47.2683319Z env: 2025-10-10T01:05:47.2683475Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2683805Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2684142Z DEVICE_NAME: 2025-10-10T01:05:47.2684319Z DEVICE_TYPE: 2025-10-10T01:05:47.2684611Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390 2025-10-10T01:05:47.2684948Z ##[endgroup] 2025-10-10T01:05:47.2787161Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T01:05:47.2787413Z with: 2025-10-10T01:05:47.2787589Z s3-bucket: gha-artifacts 2025-10-10T01:05:47.2787831Z s3-prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:47.2788138Z retention-days: 14 2025-10-10T01:05:47.2788362Z if-no-files-found: warn 2025-10-10T01:05:47.2788603Z path: test-jsons-*.zip 2025-10-10T01:05:47.2788833Z name: artifact 2025-10-10T01:05:47.2789001Z region: us-east-1 2025-10-10T01:05:47.2789191Z env: 2025-10-10T01:05:47.2789374Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.2789738Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.2790087Z DEVICE_NAME: 2025-10-10T01:05:47.2790257Z DEVICE_TYPE: 2025-10-10T01:05:47.2790431Z ##[endgroup] 2025-10-10T01:05:47.5504289Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T01:05:47.5504653Z With the provided path, there will be 1 file uploaded 2025-10-10T01:05:47.5504991Z Uploading to s3 prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:47.5537917Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:47.7238688Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:47.7404759Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T01:05:47.7405016Z with: 2025-10-10T01:05:47.7405204Z s3-bucket: gha-artifacts 2025-10-10T01:05:47.7405466Z s3-prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:47.7405729Z retention-days: 14 2025-10-10T01:05:47.7405940Z if-no-files-found: error 2025-10-10T01:05:47.7406166Z path: test-reports-*.zip 2025-10-10T01:05:47.7406377Z name: artifact 2025-10-10T01:05:47.7406566Z region: us-east-1 2025-10-10T01:05:47.7406745Z env: 2025-10-10T01:05:47.7406920Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:47.7407265Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:47.7407613Z DEVICE_NAME: 2025-10-10T01:05:47.7407800Z DEVICE_TYPE: 2025-10-10T01:05:47.7407974Z ##[endgroup] 2025-10-10T01:05:48.0191254Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T01:05:48.0196901Z With the provided path, there will be 1 file uploaded 2025-10-10T01:05:48.0201340Z Uploading to s3 prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:48.0220059Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:48.1486537Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:48.1646531Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T01:05:48.1646810Z with: 2025-10-10T01:05:48.1646988Z s3-bucket: gha-artifacts 2025-10-10T01:05:48.1647242Z s3-prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:48.1647569Z retention-days: 14 2025-10-10T01:05:48.1647759Z if-no-files-found: ignore 2025-10-10T01:05:48.1647956Z path: logs-*.zip 2025-10-10T01:05:48.1648128Z name: artifact 2025-10-10T01:05:48.1648292Z region: us-east-1 2025-10-10T01:05:48.1648461Z env: 2025-10-10T01:05:48.1648624Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:48.1648966Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:48.1649308Z DEVICE_NAME: 2025-10-10T01:05:48.1649481Z DEVICE_TYPE: 2025-10-10T01:05:48.1649707Z ##[endgroup] 2025-10-10T01:05:48.4257639Z NOTE: s3-prefix specified, ignoring name parameter 2025-10-10T01:05:48.4258205Z With the provided path, there will be 1 file uploaded 2025-10-10T01:05:48.4258703Z Uploading to s3 prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:48.4291856Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:48.5440963Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:48.5597774Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-10-10T01:05:48.5598020Z with: 2025-10-10T01:05:48.5598203Z s3-bucket: gha-artifacts 2025-10-10T01:05:48.5598448Z s3-prefix: pytorch/pytorch/18392306145/1/artifact 2025-10-10T01:05:48.5598709Z retention-days: 14 2025-10-10T01:05:48.5598894Z if-no-files-found: ignore 2025-10-10T01:05:48.5599094Z path: debug-*.zip 2025-10-10T01:05:48.5599273Z name: artifact 2025-10-10T01:05:48.5599446Z region: us-east-1 2025-10-10T01:05:48.5599609Z env: 2025-10-10T01:05:48.5599776Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:48.5600115Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:48.5600452Z DEVICE_NAME: 2025-10-10T01:05:48.5600619Z DEVICE_TYPE: 2025-10-10T01:05:48.5600791Z ##[endgroup] 2025-10-10T01:05:48.8279264Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-10-10T01:05:48.8450947Z ##[group]Run # shellcheck disable=SC2156 2025-10-10T01:05:48.8451211Z # shellcheck disable=SC2156 2025-10-10T01:05:48.8451559Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-10-10T01:05:48.8456604Z shell: /usr/bin/bash -e {0} 2025-10-10T01:05:48.8456794Z env: 2025-10-10T01:05:48.8456957Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:48.8457251Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:48.8457560Z DEVICE_NAME: 2025-10-10T01:05:48.8457720Z DEVICE_TYPE: 2025-10-10T01:05:48.8457880Z ##[endgroup] 2025-10-10T01:05:49.0303044Z Prepare all required actions 2025-10-10T01:05:49.0303411Z Getting action download info 2025-10-10T01:05:49.1564709Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-10-10T01:05:49.1564979Z with: 2025-10-10T01:05:49.1565150Z job_id: 52406535390 2025-10-10T01:05:49.1565472Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T01:05:49.1565832Z workflow_name: inductor 2025-10-10T01:05:49.1566022Z workflow_run_id: 18392306145 2025-10-10T01:05:49.1566224Z workflow_attempt: 1 2025-10-10T01:05:49.1566397Z env: 2025-10-10T01:05:49.1566547Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:49.1566875Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:49.1567201Z DEVICE_NAME: 2025-10-10T01:05:49.1567373Z DEVICE_TYPE: 2025-10-10T01:05:49.1567541Z ##[endgroup] 2025-10-10T01:05:49.1590749Z ##[group]Run echo "workflow_id: 18392306145" 2025-10-10T01:05:49.1591072Z echo "workflow_id: 18392306145" 2025-10-10T01:05:49.1591321Z echo "workflow_attempt: 1" 2025-10-10T01:05:49.1591549Z echo "workflow_Name: inductor" 2025-10-10T01:05:49.1591782Z echo "job_id: 52406535390" 2025-10-10T01:05:49.1592219Z echo "job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-10-10T01:05:49.1592682Z echo "artifact_prefix: " 2025-10-10T01:05:49.1592909Z python3 --version 2025-10-10T01:05:49.1597772Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:49.1598040Z env: 2025-10-10T01:05:49.1598217Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:49.1598553Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:49.1598889Z DEVICE_NAME: 2025-10-10T01:05:49.1599071Z DEVICE_TYPE: 2025-10-10T01:05:49.1599248Z ##[endgroup] 2025-10-10T01:05:49.1621038Z workflow_id: 18392306145 2025-10-10T01:05:49.1621420Z workflow_attempt: 1 2025-10-10T01:05:49.1621619Z workflow_Name: inductor 2025-10-10T01:05:49.1621822Z job_id: 52406535390 2025-10-10T01:05:49.1622177Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-10-10T01:05:49.1622547Z artifact_prefix: 2025-10-10T01:05:49.1636563Z Python 3.9.23 2025-10-10T01:05:49.1676538Z ##[group]Run nick-fields/retry@v3.0.0 2025-10-10T01:05:49.1676781Z with: 2025-10-10T01:05:49.1676960Z shell: bash 2025-10-10T01:05:49.1677143Z timeout_minutes: 5 2025-10-10T01:05:49.1677328Z max_attempts: 5 2025-10-10T01:05:49.1677523Z retry_wait_seconds: 30 2025-10-10T01:05:49.1677928Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-10-10T01:05:49.1678346Z polling_interval_seconds: 1 2025-10-10T01:05:49.1678562Z warning_on_retry: true 2025-10-10T01:05:49.1678758Z continue_on_error: false 2025-10-10T01:05:49.1678958Z env: 2025-10-10T01:05:49.1679147Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:49.1679487Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:49.1679834Z DEVICE_NAME: 2025-10-10T01:05:49.1680017Z DEVICE_TYPE: 2025-10-10T01:05:49.1680197Z ##[endgroup] 2025-10-10T01:05:49.4448760Z Defaulting to user installation because normal site-packages is not writeable 2025-10-10T01:05:49.5097711Z Collecting python-dateutil==2.8.2 2025-10-10T01:05:49.5238344Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-10-10T01:05:50.2379611Z Collecting boto3==1.35.42 2025-10-10T01:05:50.2441681Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-10-10T01:05:50.6503422Z Collecting pandas==2.1.3 2025-10-10T01:05:50.6547456Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-10-10T01:05:50.8248217Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-10-10T01:05:50.8262126Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-10-10T01:05:50.8294590Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-10-10T01:05:50.8300827Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-10-10T01:05:50.8306042Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-10-10T01:05:51.5012004Z Collecting numpy<2,>=1.22.4 2025-10-10T01:05:51.5048374Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-10-10T01:05:51.6262213Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-10-10T01:05:51.6563705Z Collecting tzdata>=2022.1 2025-10-10T01:05:51.6603981Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-10-10T01:05:51.6743520Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-10-10T01:05:51.6744988Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-10-10T01:05:51.6788876Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-10-10T01:05:51.6887172Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-10-10T01:05:51.6971827Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-10-10T01:05:51.6972788Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0) 2025-10-10T01:05:51.8499724Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-10-10T01:05:55.9480189Z Attempting uninstall: boto3 2025-10-10T01:05:55.9480514Z Found existing installation: boto3 1.35.33 2025-10-10T01:05:55.9547454Z Uninstalling boto3-1.35.33: 2025-10-10T01:05:55.9559721Z Successfully uninstalled boto3-1.35.33 2025-10-10T01:05:56.0004511Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-10-10T01:05:56.2397516Z Command completed after 1 attempt(s). 2025-10-10T01:05:56.2457975Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-10-10T01:05:56.2458405Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-10-10T01:05:56.2458745Z  --workflow-run-id "18392306145" \ 2025-10-10T01:05:56.2458998Z  --workflow-name "inductor" \ 2025-10-10T01:05:56.2459221Z  --workflow-run-attempt "1" \ 2025-10-10T01:05:56.2459433Z  --job-id "52406535390" \ 2025-10-10T01:05:56.2459768Z  --job-name "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-10-10T01:05:56.2460116Z  --local-path "" \ 2025-10-10T01:05:56.2460314Z  --artifact-prefix "" 2025-10-10T01:05:56.2466500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:56.2466766Z env: 2025-10-10T01:05:56.2466941Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:56.2467370Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:56.2467707Z DEVICE_NAME: 2025-10-10T01:05:56.2467888Z DEVICE_TYPE: 2025-10-10T01:05:56.2468065Z ##[endgroup] 2025-10-10T01:05:57.0567483Z repo: pytorch/pytorch 2025-10-10T01:05:57.0572337Z Search for test log in s3 bucket: ossci-utilization 2025-10-10T01:05:57.0572760Z Downloading logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:57.0578490Z extracting usage_log.txt from zip file logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_52406535390.zip 2025-10-10T01:05:57.0580850Z Converted Log Model: UtilizationMetadata: 2025-10-10T01:05:57.0581707Z UtilizationMetadata(level='metadata', workflow_id='18392306145', job_id='52406535390', workflow_name='inductor', job_name='inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1760057343, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-10-10T01:05:57.0590711Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-10-10T01:05:57.0592818Z [db model] Peek db timeseries 2025-10-10T01:05:57.0593212Z :{ 2025-10-10T01:05:57.0593457Z "created_at": 1760058356, 2025-10-10T01:05:57.0593764Z "type": "utilization", 2025-10-10T01:05:57.0594100Z "tags": [ 2025-10-10T01:05:57.0594366Z "record" 2025-10-10T01:05:57.0594624Z ], 2025-10-10T01:05:57.0594879Z "time_stamp": 1760057343, 2025-10-10T01:05:57.0595301Z "repo": "pytorch/pytorch", 2025-10-10T01:05:57.0595524Z "workflow_id": 18392306145, 2025-10-10T01:05:57.0595735Z "run_attempt": 1, 2025-10-10T01:05:57.0596153Z "job_id": 52406535390, 2025-10-10T01:05:57.0596367Z "workflow_name": "inductor", 2025-10-10T01:05:57.0596736Z "job_name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-10-10T01:05:57.0597063Z "json_data": "{}" 2025-10-10T01:05:57.0597229Z } 2025-10-10T01:05:57.0597543Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/18392306145/1/52406535390/metadata 2025-10-10T01:05:57.0598081Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/18392306145/1/52406535390/metadata 2025-10-10T01:05:57.0598695Z Writing 197 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/18392306145/1/52406535390/time_series 2025-10-10T01:05:57.0599315Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/18392306145/1/52406535390/time_series 2025-10-10T01:05:57.1645193Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-10-10T01:05:57.1645493Z with: 2025-10-10T01:05:57.1645641Z env: 2025-10-10T01:05:57.1645790Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:57.1646096Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:57.1646406Z DEVICE_NAME: 2025-10-10T01:05:57.1646578Z DEVICE_TYPE: 2025-10-10T01:05:57.1646728Z ##[endgroup] 2025-10-10T01:05:57.1664941Z ##[group]Run set -eou pipefail 2025-10-10T01:05:57.1665412Z set -eou pipefail 2025-10-10T01:05:57.1665612Z  2025-10-10T01:05:57.1665879Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-10-10T01:05:57.1666196Z for _ in $(seq 1440); do 2025-10-10T01:05:57.1666707Z  # Break if no ssh session exists anymore 2025-10-10T01:05:57.1666964Z  if [ "$(who)" = "" ]; then 2025-10-10T01:05:57.1667179Z  break 2025-10-10T01:05:57.1667411Z  fi 2025-10-10T01:05:57.1667590Z  echo "." 2025-10-10T01:05:57.1667780Z  sleep 5 2025-10-10T01:05:57.1667962Z done 2025-10-10T01:05:57.1672440Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:57.1672680Z env: 2025-10-10T01:05:57.1672833Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:57.1673202Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:57.1673509Z DEVICE_NAME: 2025-10-10T01:05:57.1673673Z DEVICE_TYPE: 2025-10-10T01:05:57.1673829Z ##[endgroup] 2025-10-10T01:05:57.1693371Z Holding runner for 2 hours until all ssh sessions have logged out 2025-10-10T01:05:57.1784936Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T01:05:57.1785287Z # ignore expansion of "docker ps -q" since it could be empty 2025-10-10T01:05:57.1785559Z # shellcheck disable=SC2046 2025-10-10T01:05:57.1785798Z docker stop $(docker ps -q) || true 2025-10-10T01:05:57.1786019Z # Prune all of the docker images 2025-10-10T01:05:57.1786238Z docker system prune -af 2025-10-10T01:05:57.1790541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:05:57.1790775Z env: 2025-10-10T01:05:57.1790927Z GIT_DEFAULT_BRANCH: main 2025-10-10T01:05:57.1791212Z DOCKER_CONTAINER_ID: 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:05:57.1791509Z DEVICE_NAME: 2025-10-10T01:05:57.1791665Z DEVICE_TYPE: 2025-10-10T01:05:57.1791820Z ##[endgroup] 2025-10-10T01:06:08.1740911Z 3283ba949f9d 2025-10-10T01:06:08.4961311Z Deleted Containers: 2025-10-10T01:06:08.4961708Z 3283ba949f9dbea3e271af3ce027ff8981538bf64a794955dad568ddfcf0392c 2025-10-10T01:06:08.4962027Z 2025-10-10T01:06:15.8510286Z Deleted Images: 2025-10-10T01:06:15.8511009Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-d8be0384e085f551506bd739678109fa0f5ee7ac 2025-10-10T01:06:15.8511902Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:2e6e7297c9105722429dc16c6731f2c79bd9d6b81eb7a39feebebd90cfe1676f 2025-10-10T01:06:15.8512921Z deleted: sha256:be8feb4482de208ab33ff6be61032fcdbeafec87bebc633426f293b292801f69 2025-10-10T01:06:15.8513377Z deleted: sha256:32717f525a3d285caa7bc2f1cd88830e49fd4771a01cde6dac3e27ae39ecc6f6 2025-10-10T01:06:15.8513832Z deleted: sha256:89099a0da699daa509b92ebee4e89f768b46f1ceaed001c87506de00879d3e37 2025-10-10T01:06:15.8514278Z deleted: sha256:5a61a2b1b46bc2f9a60d8df72058a77901b4653964ae9bcc36f1515cb8901a67 2025-10-10T01:06:15.8514726Z deleted: sha256:b52c6bcff4a604be190679303bccbcbb6dc2af52ba8023006c10fb5f8a796bc3 2025-10-10T01:06:15.8515155Z deleted: sha256:dc3006fe1f308bdc2c71420314491a93aae4f258f1fa66498cbba80e9884fd61 2025-10-10T01:06:15.8515584Z deleted: sha256:5f2dea2c86d0735f409c1aed288265c8fb10ceb66b3dc8010ebebb9fc3661d75 2025-10-10T01:06:15.8516149Z deleted: sha256:bd129dca614f7e28e081d815314995a1836f6013972a7e6f81e71f27db85f5cc 2025-10-10T01:06:15.8516565Z deleted: sha256:3c1cffd2e9c1e90a623e85034a5a340754a8ffd002fcc106b370b075d568c5aa 2025-10-10T01:06:15.8516983Z deleted: sha256:67643d632829762a8fa1c87034cccd913ff07879f94edabb92ae83e5f4f06f37 2025-10-10T01:06:15.8517420Z deleted: sha256:e7fa85a36f9e4d7c86706e7599534d88eed6027b4863e3e8182a41bdef5c8cca 2025-10-10T01:06:15.8517847Z deleted: sha256:6029e794746fedfade39b53f444cd9f29bfbbc1ac382706c9bb2518e510dd969 2025-10-10T01:06:15.8518263Z deleted: sha256:4643060d6754011aba6906d42be1247ade2a19627412b4009282307d6d6c3ad6 2025-10-10T01:06:15.8518669Z deleted: sha256:33310bc41c5b8b726b549dbe76b11728e62e5a3a5057ce3fd0f19431e9212401 2025-10-10T01:06:15.8519088Z deleted: sha256:5c34eec5d645b307d88c7bc8759c858d957504a21dd6445322d5c49cf3525a62 2025-10-10T01:06:15.8519505Z deleted: sha256:2f5165a0705417e314bf72114ff846365e3ab3fe843d8b3fc18a9bf494a005aa 2025-10-10T01:06:15.8519924Z deleted: sha256:395668f7a97e6a8502b4cb3f7e18e33a9f0f4f9ec0446a225b4d44e5fab84eb1 2025-10-10T01:06:15.8520342Z deleted: sha256:a25d3f2697abf5b202ad996ea2ab0fcc49991850b816d9b35e4a40a4a4751e73 2025-10-10T01:06:15.8520766Z deleted: sha256:7540cf9f829048e2e741f56443a32a2adcbcc92c8c915e6e39380226455cb2a3 2025-10-10T01:06:15.8521190Z deleted: sha256:e09417614e19acfd72eef94af5fc6f4738d679aeeb5e48c5e23d7d65ac750dbb 2025-10-10T01:06:15.8521614Z deleted: sha256:d214215a5fe7e25194d83be16eb8a41e2d2df56543a7f74c59261c1c035b6ce9 2025-10-10T01:06:15.8522123Z deleted: sha256:22d5ddaa5c80746ee7248c979900026e7d9bcad7edca5e404c28bdf7bd1d2d66 2025-10-10T01:06:15.8522548Z deleted: sha256:56c2dfda0710be12fa50d1b1fab7b9dd71fc356dc1703c45c76ffd62ea3bad97 2025-10-10T01:06:15.8523028Z deleted: sha256:511b946c468c4dbe00e2721ecead37e0faa77881bbb4c5efd216caac96fe9adb 2025-10-10T01:06:15.8523455Z deleted: sha256:91190044981e87b8a8a469eb84b3495c1ca7aac3f201ab7461d75fc52fdfb885 2025-10-10T01:06:15.8523878Z deleted: sha256:b3e6c410177e555ff0e08704af0dedd546a92e3da0c4552f2086d877187b7835 2025-10-10T01:06:15.8524311Z deleted: sha256:9dee748523d58aa24cce2eba638c5750a27719097dda1b513e8caff5d2a68321 2025-10-10T01:06:15.8524737Z deleted: sha256:f9da7acef3bdd3c4fe715d82796f20ea41c6289fa6073b9a605be627bd5a562c 2025-10-10T01:06:15.8525171Z deleted: sha256:adff21a51340aadaae2e7e8f6c7369348cfe1fd75a50c7cff650724ccb79fcc2 2025-10-10T01:06:15.8525600Z deleted: sha256:007c740f1521c4ffb9fdd2d81848a4b09621b0a8e7e8a92402ac2f47bb08a7b7 2025-10-10T01:06:15.8526023Z deleted: sha256:76a91430459cf74b6bc295ed6bb0c328a34b9d78b127f09d373baeb47ee128aa 2025-10-10T01:06:15.8526453Z deleted: sha256:69639d06725bdeea11974792fe1a0a2f8ebfb8e6d9390c253bbda01116cc35f8 2025-10-10T01:06:15.8526881Z deleted: sha256:0dde43a8e27b6bbe633ca4b996db7933a8df61a3cae05488b9ee28a7118a8e9c 2025-10-10T01:06:15.8527328Z deleted: sha256:067e0bfa5debcfb451c1a1cdbe0b2e0d8b6f59434187c111fa827dd30edaa196 2025-10-10T01:06:15.8527758Z deleted: sha256:a40880c68bc6527b3e299c7206169388a3e0d96f8da39df923560ab34e77a9bd 2025-10-10T01:06:15.8528181Z deleted: sha256:35b51ddfb7d6d1960f2015df6a104f10de935595754fa7b1230b3c9614d34a1e 2025-10-10T01:06:15.8528616Z deleted: sha256:d2dd9c2b7710ec5c6b9119c95dabc086ecab34d252861a0c468917983025b30c 2025-10-10T01:06:15.8529014Z deleted: sha256:74672ed2f435e3580f68d17aacd136e3fb5413a63d8866b8912bae95296918af 2025-10-10T01:06:15.8529420Z deleted: sha256:572d1f2e41cfe9e797d8d3171ddba1825d8235b7bc0138e561a1bfe368f50e74 2025-10-10T01:06:15.8529826Z deleted: sha256:9d7624a1f32975ece361d9560c355727a398c8d82c1c60c2bfff0cc7a50c636b 2025-10-10T01:06:15.8530231Z deleted: sha256:53ec3fd4cb54752db3cd12bc186306aef5243a988328d045ba3f028671a0c199 2025-10-10T01:06:15.8530656Z deleted: sha256:5d30e69233c4798d235aaeedbb5ac9c045881c5ba48c1e9bfdd7a0fb304c7109 2025-10-10T01:06:15.8531092Z deleted: sha256:4896ccc7f6dee8062d0afd01186a43435bbe4cae1c5ef1464bbfab0eec27e065 2025-10-10T01:06:15.8531803Z deleted: sha256:d7687ef2b05c946f4534046fbd43e629c708eccdfa46fd5bea558faea1816507 2025-10-10T01:06:15.8532320Z deleted: sha256:11e8b4b9d637ff11aa4a3ed324259f844f62474a7bb6d190853fd4246a6bd085 2025-10-10T01:06:15.8532744Z deleted: sha256:20c0264709a6c9fa1faea5cedb7b28e267e8078ebac2f2ef71cf198a0b632d9c 2025-10-10T01:06:15.8533160Z deleted: sha256:930497a4e10935158418e96eb34d1efa4eebe099d1f94d64fb7554ffddac3693 2025-10-10T01:06:15.8533570Z deleted: sha256:a89c0a03f778ac7258bc59bf2516808a3ef8461f06afd28e2ea46879cae86f37 2025-10-10T01:06:15.8533988Z deleted: sha256:867362c9c083f1215fd021993c8fd9788ba092a66b15064795b7f30bbc8fc3dd 2025-10-10T01:06:15.8534420Z deleted: sha256:94ecb8938ae84bcbdce38d2cac58e922e1e7d6ca89160419825a78ca4a0cf4df 2025-10-10T01:06:15.8534856Z deleted: sha256:229122817f615c3ddf4043c3f43caed1740ab296e5a71c45cd498d3c7549ab02 2025-10-10T01:06:15.8535297Z deleted: sha256:a1e1913a740d1a2dbe501b7bdffc9f2103803e727a8763556311c237066d7757 2025-10-10T01:06:15.8535722Z deleted: sha256:4f1aac1828f91dcb2cf1b7f84ab4870332748ff15a23f32f3d6631e61842db56 2025-10-10T01:06:15.8536148Z deleted: sha256:1174854b31919c2f2c855f1d0b2b9c9146554fc2fe08c6c52133f8229e2a13e8 2025-10-10T01:06:15.8536564Z deleted: sha256:0084d797a0f3e8745bcf730e59cf282517421cb3ea908bb42b6c6460b688dc1c 2025-10-10T01:06:15.8536979Z deleted: sha256:69d9949cd5f32f0e6258b45daade0f007e1e3a183e316cda52ed106c7155bc7f 2025-10-10T01:06:15.8537390Z deleted: sha256:514958890f92a13832d7466c07459651e67187d2b3d5f622f72873658547c606 2025-10-10T01:06:15.8537806Z deleted: sha256:dba5c0a5206561bb429d7d42eed7a460891caf60c8fa21d572717c881d288dc1 2025-10-10T01:06:15.8538258Z deleted: sha256:1dae66c5b1743ea34b634eb48ca9a0ec343b24fd950c13a9241a73d68fdd4cac 2025-10-10T01:06:15.8538678Z deleted: sha256:587ebfd38fa12c5fa21a79afa1ec2fc9e45c3700fbd5ce2f7e3e281fa3bf48d7 2025-10-10T01:06:15.8539097Z deleted: sha256:13d312f092051ae4427f9cc636a040230820a25e3e20769975cc80b76a75bbb5 2025-10-10T01:06:15.8539513Z deleted: sha256:24e4131c7344cfb6de33e062e8db0270f0764e5842a8ce1627661115a56a7f92 2025-10-10T01:06:15.8539928Z deleted: sha256:cb517e78b183b647633c2226842521e1f0e3c6ba0ff1da28253b4630286ca6a7 2025-10-10T01:06:15.8540354Z deleted: sha256:16a436ab43d9f05eb8cd4fc699b7ed1526c5cd416bd8b96c5086eab2102d133c 2025-10-10T01:06:15.8540772Z deleted: sha256:6869c280fe435918c266dd4aeb2966e37a1569d83ddb848b93293c8e86a949a0 2025-10-10T01:06:15.8541193Z deleted: sha256:674d313f2e59dcdd66dbc1637c9ae437863725322bce108998a11d9b0ce1b595 2025-10-10T01:06:15.8541614Z deleted: sha256:98014cbe47b24acd1e13ab5b692942b0618247b9c6be910285ae04e7a8a88198 2025-10-10T01:06:15.8542042Z deleted: sha256:bccee968e0721414dd50b98c394e14e1a94abff32ad70300e436d768fb4d4ec9 2025-10-10T01:06:15.8542465Z deleted: sha256:c12d7a7c2855250aa7b7b96a306bd116affa640b86c1e593673710f085f1220d 2025-10-10T01:06:15.8542877Z deleted: sha256:635ddff3f3e03b85f2fa2a22e3c16854c8959545871f8652f501a3d3ab1dc714 2025-10-10T01:06:15.8543308Z deleted: sha256:b5ddbb0d5175429b14b1a61ab3efcb06c08087aca2bcaf8dfe010e2c23f76534 2025-10-10T01:06:15.8543734Z deleted: sha256:80286bfd89cdbd44683c8e1109caa279992c9073fc6a15bbd3310c4016cf4bf8 2025-10-10T01:06:15.8544161Z deleted: sha256:2c6043e15836cbb385136f3cee4deb4eb40b626fe37f2a64eef27f7d71c72c0b 2025-10-10T01:06:15.8544582Z deleted: sha256:0a08a83150c724f62243256de60fe955029a742dbfac10604df71a0c73d53240 2025-10-10T01:06:15.8545018Z deleted: sha256:fdeed8b36387087d16c1b6ef965477a5bd10a70984620de242efa59488510d46 2025-10-10T01:06:15.8545440Z deleted: sha256:4325ea5e69fbd05c6eb48335ba1835e7a44cff96f08dc62216a01d54d1454bf5 2025-10-10T01:06:15.8545860Z deleted: sha256:767e56ba346ae714b6e6b816baa839051145ed78cfa0e4524a86cc287b0c4b00 2025-10-10T01:06:15.8546101Z 2025-10-10T01:06:15.8546210Z Total reclaimed space: 53.31GB 2025-10-10T01:06:15.8628853Z Post job cleanup. 2025-10-10T01:06:15.8677394Z Post job cleanup. 2025-10-10T01:06:15.9592514Z [command]/usr/bin/git version 2025-10-10T01:06:15.9632744Z git version 2.50.1 2025-10-10T01:06:15.9663465Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/c9fc4153-502a-459d-bd7b-f5f9ce2ad638/.gitconfig' 2025-10-10T01:06:15.9686233Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c9fc4153-502a-459d-bd7b-f5f9ce2ad638' before making global git config changes 2025-10-10T01:06:15.9687043Z Adding repository directory to the temporary git global config as a safe directory 2025-10-10T01:06:15.9688144Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-10-10T01:06:15.9732120Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-10-10T01:06:15.9767573Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-10-10T01:06:16.0084301Z Entering 'android/libs/fbjni' 2025-10-10T01:06:16.0142095Z Entering 'third_party/FP16' 2025-10-10T01:06:16.0198226Z Entering 'third_party/FXdiv' 2025-10-10T01:06:16.0258070Z Entering 'third_party/NNPACK' 2025-10-10T01:06:16.0315367Z Entering 'third_party/NVTX' 2025-10-10T01:06:16.0372295Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:06:16.0427986Z Entering 'third_party/XNNPACK' 2025-10-10T01:06:16.0496022Z Entering 'third_party/aiter' 2025-10-10T01:06:16.0556637Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:06:16.0618173Z Entering 'third_party/benchmark' 2025-10-10T01:06:16.0679509Z Entering 'third_party/composable_kernel' 2025-10-10T01:06:16.0739838Z Entering 'third_party/cpp-httplib' 2025-10-10T01:06:16.0794160Z Entering 'third_party/cpuinfo' 2025-10-10T01:06:16.0861801Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:06:16.0918058Z Entering 'third_party/cutlass' 2025-10-10T01:06:16.0986301Z Entering 'third_party/fbgemm' 2025-10-10T01:06:16.1042815Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:06:16.1095292Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:06:16.1159858Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:06:16.1213185Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:06:16.1281535Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:06:16.1335638Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:06:16.1391882Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:06:16.1456071Z Entering 'third_party/flash-attention' 2025-10-10T01:06:16.1504725Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:06:16.1569752Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:06:16.1627945Z Entering 'third_party/flatbuffers' 2025-10-10T01:06:16.1686278Z Entering 'third_party/fmt' 2025-10-10T01:06:16.1738277Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:06:16.1796232Z Entering 'third_party/gloo' 2025-10-10T01:06:16.1851066Z Entering 'third_party/googletest' 2025-10-10T01:06:16.1906196Z Entering 'third_party/ideep' 2025-10-10T01:06:16.1962069Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:06:16.2024908Z Entering 'third_party/ittapi' 2025-10-10T01:06:16.2088618Z Entering 'third_party/kineto' 2025-10-10T01:06:16.2143780Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:06:16.2200032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:06:16.2257534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:06:16.2310338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:06:16.2361271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:06:16.2413994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:06:16.2474820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:06:16.2529106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:06:16.2583498Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:06:16.2633282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:06:16.2692483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:06:16.2745345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:06:16.2801503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:06:16.2859145Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:06:16.2911995Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:06:16.2974465Z Entering 'third_party/kleidiai' 2025-10-10T01:06:16.3027665Z Entering 'third_party/mimalloc' 2025-10-10T01:06:16.3090658Z Entering 'third_party/nlohmann' 2025-10-10T01:06:16.3146356Z Entering 'third_party/onnx' 2025-10-10T01:06:16.3212249Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:06:16.3274113Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:06:16.3327420Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:06:16.3385071Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:06:16.3447739Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:06:16.3498192Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:06:16.3552917Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:06:16.3607732Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:06:16.3662793Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:06:16.3715366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:06:16.3777838Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:06:16.3830664Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:06:16.3903792Z Entering 'third_party/pocketfft' 2025-10-10T01:06:16.3959675Z Entering 'third_party/protobuf' 2025-10-10T01:06:16.4015593Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:06:16.4078377Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:06:16.4137701Z Entering 'third_party/psimd' 2025-10-10T01:06:16.4193543Z Entering 'third_party/pthreadpool' 2025-10-10T01:06:16.4250203Z Entering 'third_party/pybind11' 2025-10-10T01:06:16.4303849Z Entering 'third_party/python-peachpy' 2025-10-10T01:06:16.4360261Z Entering 'third_party/sleef' 2025-10-10T01:06:16.4413914Z Entering 'third_party/tensorpipe' 2025-10-10T01:06:16.4467261Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:06:16.4520145Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:06:16.4578256Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:06:16.4631126Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:06:16.4688151Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:06:16.4767202Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-10-10T01:06:16.4793054Z http.https://github.com/.extraheader 2025-10-10T01:06:16.4801623Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-10-10T01:06:16.4832383Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-10-10T01:06:16.5156994Z Entering 'android/libs/fbjni' 2025-10-10T01:06:16.5186593Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5225540Z Entering 'third_party/FP16' 2025-10-10T01:06:16.5263709Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5299945Z Entering 'third_party/FXdiv' 2025-10-10T01:06:16.5336091Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5377276Z Entering 'third_party/NNPACK' 2025-10-10T01:06:16.5410483Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5450532Z Entering 'third_party/NVTX' 2025-10-10T01:06:16.5482754Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5517591Z Entering 'third_party/VulkanMemoryAllocator' 2025-10-10T01:06:16.5560321Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5589122Z Entering 'third_party/XNNPACK' 2025-10-10T01:06:16.5620807Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5673676Z Entering 'third_party/aiter' 2025-10-10T01:06:16.5705703Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5744754Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-10-10T01:06:16.5784724Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5827373Z Entering 'third_party/benchmark' 2025-10-10T01:06:16.5865709Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5904331Z Entering 'third_party/composable_kernel' 2025-10-10T01:06:16.5944680Z http.https://github.com/.extraheader 2025-10-10T01:06:16.5986937Z Entering 'third_party/cpp-httplib' 2025-10-10T01:06:16.6020447Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6060890Z Entering 'third_party/cpuinfo' 2025-10-10T01:06:16.6095974Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6131367Z Entering 'third_party/cudnn_frontend' 2025-10-10T01:06:16.6171908Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6210734Z Entering 'third_party/cutlass' 2025-10-10T01:06:16.6247755Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6288662Z Entering 'third_party/fbgemm' 2025-10-10T01:06:16.6326897Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6365640Z Entering 'third_party/fbgemm/external/asmjit' 2025-10-10T01:06:16.6400101Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6441034Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-10-10T01:06:16.6473397Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6514653Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-10-10T01:06:16.6553228Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6594976Z Entering 'third_party/fbgemm/external/cutlass' 2025-10-10T01:06:16.6625505Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6677089Z Entering 'third_party/fbgemm/external/googletest' 2025-10-10T01:06:16.6703889Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6740721Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-10-10T01:06:16.6777427Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6814883Z Entering 'third_party/fbgemm/external/json' 2025-10-10T01:06:16.6855754Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6894004Z Entering 'third_party/flash-attention' 2025-10-10T01:06:16.6931037Z http.https://github.com/.extraheader 2025-10-10T01:06:16.6969373Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-10-10T01:06:16.7005297Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7050242Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-10-10T01:06:16.7079650Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7126436Z Entering 'third_party/flatbuffers' 2025-10-10T01:06:16.7163684Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7205564Z Entering 'third_party/fmt' 2025-10-10T01:06:16.7243371Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7282365Z Entering 'third_party/gemmlowp/gemmlowp' 2025-10-10T01:06:16.7314494Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7358596Z Entering 'third_party/gloo' 2025-10-10T01:06:16.7395652Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7429150Z Entering 'third_party/googletest' 2025-10-10T01:06:16.7464277Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7495165Z Entering 'third_party/ideep' 2025-10-10T01:06:16.7530459Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7567712Z Entering 'third_party/ideep/mkl-dnn' 2025-10-10T01:06:16.7600877Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7642312Z Entering 'third_party/ittapi' 2025-10-10T01:06:16.7674117Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7713096Z Entering 'third_party/kineto' 2025-10-10T01:06:16.7747277Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7778648Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-10-10T01:06:16.7815217Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7849088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-10-10T01:06:16.7890165Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7922158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-10-10T01:06:16.7958964Z http.https://github.com/.extraheader 2025-10-10T01:06:16.7996132Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-10-10T01:06:16.8036954Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8070195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-10-10T01:06:16.8104904Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8138688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-10-10T01:06:16.8170827Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8209076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-10-10T01:06:16.8243843Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8283444Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-10-10T01:06:16.8317293Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8355193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-10-10T01:06:16.8390290Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8423761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-10-10T01:06:16.8464624Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8501136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-10-10T01:06:16.8537870Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8573452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:06:16.8612463Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8650876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:06:16.8692080Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8731118Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-10-10T01:06:16.8768031Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8802630Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-10-10T01:06:16.8844615Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8879224Z Entering 'third_party/kleidiai' 2025-10-10T01:06:16.8910444Z http.https://github.com/.extraheader 2025-10-10T01:06:16.8946026Z Entering 'third_party/mimalloc' 2025-10-10T01:06:16.8981959Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9019588Z Entering 'third_party/nlohmann' 2025-10-10T01:06:16.9056753Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9092436Z Entering 'third_party/onnx' 2025-10-10T01:06:16.9127304Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9175075Z Entering 'third_party/onnx/third_party/pybind11' 2025-10-10T01:06:16.9212506Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9250841Z Entering 'third_party/opentelemetry-cpp' 2025-10-10T01:06:16.9284953Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9315901Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-10-10T01:06:16.9350808Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9387437Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-10-10T01:06:16.9420934Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9462579Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-10-10T01:06:16.9497103Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9529531Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-10-10T01:06:16.9568820Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9606947Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-10-10T01:06:16.9647187Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9681350Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-10-10T01:06:16.9715606Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9748344Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-10-10T01:06:16.9782582Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9818290Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-10-10T01:06:16.9853325Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9888176Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-10-10T01:06:16.9916786Z http.https://github.com/.extraheader 2025-10-10T01:06:16.9968745Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-10-10T01:06:16.9999901Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0051244Z Entering 'third_party/pocketfft' 2025-10-10T01:06:17.0087641Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0118136Z Entering 'third_party/protobuf' 2025-10-10T01:06:17.0156975Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0192867Z Entering 'third_party/protobuf/third_party/benchmark' 2025-10-10T01:06:17.0227066Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0264518Z Entering 'third_party/protobuf/third_party/googletest' 2025-10-10T01:06:17.0299413Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0338124Z Entering 'third_party/psimd' 2025-10-10T01:06:17.0367910Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0401832Z Entering 'third_party/pthreadpool' 2025-10-10T01:06:17.0443271Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0479496Z Entering 'third_party/pybind11' 2025-10-10T01:06:17.0512713Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0547087Z Entering 'third_party/python-peachpy' 2025-10-10T01:06:17.0586569Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0618634Z Entering 'third_party/sleef' 2025-10-10T01:06:17.0656950Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0697935Z Entering 'third_party/tensorpipe' 2025-10-10T01:06:17.0733508Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0773444Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-10-10T01:06:17.0811619Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0848288Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-10-10T01:06:17.0881485Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0919143Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-10-10T01:06:17.0957862Z http.https://github.com/.extraheader 2025-10-10T01:06:17.0991166Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-10-10T01:06:17.1022929Z http.https://github.com/.extraheader 2025-10-10T01:06:17.1062671Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-10-10T01:06:17.1097118Z http.https://github.com/.extraheader 2025-10-10T01:06:17.1236858Z A job completed hook has been configured by the self-hosted runner administrator 2025-10-10T01:06:17.1260788Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-10-10T01:06:17.1264413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-10-10T01:06:17.1264686Z ##[endgroup] 2025-10-10T01:06:17.1347787Z [!ALERT!] Swap in detected! [!ALERT!] 2025-10-10T01:06:26.2084170Z [!ALERT!] Swap out detected [!ALERT!] 2025-10-10T01:06:41.2987896Z Cleaning up orphan processes